zero downtime bi-directional physical to virtual …...zero downtime bi-directional physical to...
TRANSCRIPT
© 2011 VCE Company, LLC. All rights reserved. 1
ZERO DOWNTIME BI-DIRECTIONAL
PHYSICAL TO VIRTUAL MIGRATION
FOR ORACLE® EBS R12 WITH RAC
ON VBLOCK™
INFRASTRUCTURE
PLATFORMS
August 2011
VCE has built Centers of Excellence to reflect real world deployments in which solutions are developed and executed. These solutions provide VCE with insights into the challenges currently facing its customers. This paper summarizes a series of best practices that VCE, in partnership with EMC – Proven Solutions, discovered, validated, or otherwise encountered during the validation of this paper.
© 2011 VCE Company, LLC. All rights reserved. 2
Table of Contents
Solution Overview ....................................................................................................................... 4
Benefits of Deployment on Vblock Infrastructure Platforms ....................................................... 5
Scope of This Paper .................................................................................................................... 5
Audience ..................................................................................................................................... 5
Technology Overview ...................................................................................................................... 6
VCE Vblock Infrastructure Platform ............................................................................................ 6
Oracle Database 11g Release 2 Enterprise Edition ................................................................... 6
Oracle Real Application Clusters (RAC) ..................................................................................... 6
Oracle E-Business Suite Release 12 .......................................................................................... 7
EMC Unified Infrastructure Manager 2.1 .................................................................................... 9
Solution Architecture ..................................................................................................................... 10
Vblock Architecture ................................................................................................................... 11
Validation Environment ............................................................................................................. 11
Validation Methodology ............................................................................................................. 12
Validation Objectives ................................................................................................................. 12
Hardware Components ............................................................................................................. 13
Software Components ............................................................................................................... 13
Storage Layout .......................................................................................................................... 13
Migrating an Oracle RAC to a Virtual Environment (P2V) ............................................................. 15
Pre-requisites ............................................................................................................................ 15
Overview of the Steps ............................................................................................................... 15
Phase 1 – Creating the Physical Oracle RAC ........................................................................... 15
Phase 2 - Creating the Virtual Oracle RAC ............................................................................... 21
Phase 3 - Creating Additional Virtual Nodes ............................................................................. 28
Phase 4 - Removing the Physical Nodes from the Cluster ....................................................... 29
Phase 5 – Adding a Physical Node to the Oracle RAC Cluster (V2P)...................................... 34
© 2011 VCE Company, LLC. All rights reserved. 3
Using VMware vMotion to Reduce Planned Downtime ................................................................. 37
Performing Live Migration ......................................................................................................... 38
Verifying the Live Migration is Non-disruptive ........................................................................... 40
Scalability and Performance Considerations for Oracle EBS and RAC ........................................ 42
Conclusion ..................................................................................................................................... 43
Appendix A: Configuring SSH for User Equivalence ..................................................................... 44
© 2011 VCE Company, LLC. All rights reserved. 4
Executive Summary
Deployment and maintenance of Oracle® applications such as E-Business Suite (EBS) in a physical environment
is not a trivial task. Transitioning to a virtualized infrastructure can simplify management of Oracle EBS application
workloads and can facilitate delivery of the required levels of availability, performance, and scalability. The multi-
tier architecture of Oracle E-Business Suite makes it an ideal candidate for virtualization.
Today's business imperatives for IT demand that the deployment of applications, such as Oracle EBS, be
accomplished securely and safely, without affecting production environments. With the availability of Vblock™
Infrastructure Platforms, the deployment of Oracle EBS on this platform meets the most demanding customer
requirements.
Running multiple Oracle EBS components in virtual machines on the same physical Vblock platform provides
consolidation benefits such as less hardware, data center space, power consumption, and so forth, while lowering
total cost of ownership (TCO) and providing higher return on investment (ROI).
Migration of physical environments to virtual environments in the cloud-computing model can be achieved
effectively with zero downtime to the current data center environment by leveraging the virtualization technologies
available on the Vblock platform and Oracle Real Application Cluster (RAC). By creating service profiles and
service offerings with EMC®
Unified Infrastructure Manager (UIM) software on the Vblock platform, physical to
virtual environments can be provisioned quickly and easily for Oracle EBS and Real Application Cluster (RAC)
environments.
Solution Overview
This paper presents a methodical, structured approach to migrating a physical Oracle RAC to a virtual Oracle
RAC running Oracle EBS. This approach meets many of the challenges presented by virtualization of Oracle
databases and applications. The following table describes how the solution addresses the main challenges to
virtualizing Oracle EBS and similar applications.
Challenge Solution
Customers want to leverage the converged infrastructure and virtualization technologies of Vblock Infrastructure Platforms to run their tier 1 applications such as Oracle EBS in a virtualized environment. However, Oracle’s current support position for its database products running on a VMware
® platform
requires that issues be reproduced on a physical server.
This solution first demonstrates a non-disruptive (no downtime) method for migrating a physical Oracle database to VMware virtual machines. The solution then addresses performing this migration in reverse (virtual to physical) in the event that Oracle support requires an issue to be reproduced on a physical server.
Customers want to be able to combine the built-in high availability mechanisms of virtualization for Oracle EBS and Real Application Clusters (RAC) with VMware High Availability (HA) technology to guarantee very high uptime.
Oracle RAC 11g Release 2 (11g R2) is designed to be fault tolerant and to provide real-time HA with continuous database uptime. The use of VMware vMotion™ with VMware vSphere™ provides HA for Oracle EBS and RAC 11g R2 at both virtual and physical levels.
© 2011 VCE Company, LLC. All rights reserved. 5
Benefits of Deployment on Vblock Infrastructure Platforms
Because of a converged infrastructure and virtualization technologies, Vblock Infrastructure Platforms offer
significantly more flexibility, scalability, manageability, and security with better support than other infrastructure
stacks.
With a Vblock platform, customers can deploy all their current applications in virtualized server environments with
minimal (if any) modifications to their operating system, their application version, and so on. The Vblock platform
addresses important operational challenges, such as providing the ability to easily create snaps and clones,
application consistency on remote replication, and database refresh without sacrificing flexibility, openness, and
optimization. The Vblock platform provides open system architecture with flexible configuration options at cost
points that are compelling for customers.
Deployment of Oracle EBS in the virtualized environment of the Vblock platform provides the following key
benefits:
Containment of server sprawl and increased server utilization by running multiple Oracle EBS
components in virtual machines consolidated onto fewer, highly scalable, reliable enterprise-class
systems.
Dynamic migration of Oracle instances on virtual machines to other systems to perform hardware
maintenance or system changes on the current physical system, without disruption to end users.
Improved operational agility with both vertical and horizontal scaling to accommodate dynamic
workloads through virtualization.
Provisioning of new pre-configured Oracle EBS environments in minutes using VMware vSphere
templates.
Automatic detection of failed physical servers running Oracle application instances in production, and
automatic restart of failed Oracle application virtual machines with VMware HA.
Scope of This Paper
This paper presents a methodical, structured approach to non-disruptive migration of a physical Oracle RAC
database to a virtualized environment and addition of physical RAC nodes to the virtual cluster for support and
problem resolution purposes. The steps used to validate this approach are described in detail.
This paper describes how to:
Create a physical Oracle RAC and migrate it non-disruptively to a virtual environment on the Vblock
platform.
Add a physical Oracle RAC node to a cluster to meet support requirements by Oracle to re-produce
problems on a physical environment.
Use VMware vMotion™ to perform live, non-disruptive migration of virtual machines to different hosts
for planned maintenance, updates, and upgrades.
Audience
Oracle database administrators, storage architects, VMware administrators, VCE customers, and individuals
tasked with evaluating, acquiring, managing, operating, or deploying Oracle EBS in a virtualized data center
environment will benefit from reading this paper.
© 2011 VCE Company, LLC. All rights reserved. 6
Technology Overview
This solution uses the following major hardware and software components and technologies:
VCE Vblock Infrastructure Platform
Oracle Database 11g R2 Enterprise Edition
Oracle Real Application Clusters (RAC) 11g R2
Oracle E-Business Suite Release 12
EMC Unified Infrastructure Manager (UIM) 2.1
VCE Vblock Infrastructure Platform
Vblock Infrastructure Platforms provide pre-engineered, production-ready (fully tested) virtualized infrastructure
components, including the best of breed cloud offerings from Cisco, EMC, and VMware. Vblock Infrastructure
Platforms are designed and built to satisfy a broad range of specific customer implementation requirements. The
following table lists the main components of the Vblock Series 700 model MX used for validating the solution.
Layer Components
Compute Cisco 6120 Fabric Interconnects
Cisco UCS 5108 chassis
Cisco UCS B-series blade server packs
Network Cisco Nexus 5020 data center switches
Storage EMC storage
Cisco MDS Aggregation SAN switch
Virtualization VMware vSphere / vCenter Server
Management Symmetrix Management Console (SMC) 7.2
EMC PowerPath/VE™
Cisco UCS Manager
EMC Unified Infrastructure Manager (UIM)
Oracle Database 11g Release 2 Enterprise Edition
In Oracle Database 11g R2, both Oracle Automatic Storage Management (ASM) and Oracle Clusterware have
been integrated into a single set of binaries and named Oracle Grid Infrastructure. This now provides all cluster
and storage services required to run an Oracle RAC database.
Oracle Real Application Clusters (RAC)
Oracle RAC, with Oracle Database 11g Enterprise Edition, enables a single database to run across a cluster of
servers, providing fault tolerance, performance, and scalability with no application changes necessary. In line with
Oracle’s support position for Oracle products running on VMware, the solution described in this paper was
deployed using Oracle RAC 11.2.0.2.
Note: For Oracle's support policy on Oracle products running on VMware, refer to My Oracle Support Doc [ID 249212.1] in http://support.oracle.com.
© 2011 VCE Company, LLC. All rights reserved. 7
Oracle E-Business Suite Release 12
Oracle E-Business Suite architecture is a framework for multi-tiered distributed computing. In the Oracle E-
Business Suite model, services are distributed among multiple levels or tiers. A service is a process or a group of
processes that exercise some business logic and provide a particular functionality. A tier is a grouping of services
that can potentially span physical machines. In other words, a tier is a logical grouping of services that is not
limited by physical nodes or machines. Each tier can consist of one or more nodes, and each node can
accommodate more than one tier. For example, a single machine can contain database, application, and desktop
tiers; in the same way, a database can reside on one of many application servers or on a separate machine by
itself.
Figure 1. Oracle E-Business Suite Tiers
Desktop Tier
The desktop tier is normally the client web browser. The user interface for the desktop tier is provided through
HTML for self-service applications and a Java™ applet for forms. The desktop client is installed on demand upon
first use and then stored locally for future use. The forms client applet is packaged as JAR files and mainly
represents the presentation layer of Oracle forms. The JAR files are also downloaded on the desktop during first
use. The forms client must run within a Java Virtual Machine (JVM) and is supplied by Oracle JInitiator.
© 2011 VCE Company, LLC. All rights reserved. 8
Application Tier
The application tier hosts one or more servers that process the business logic. It communicates between the
desktop tier and database. The application tier, also commonly known as the middle tier, includes the following
major components: Web Server, Forms Server, Concurrent Processing Server, Business Intelligence System,
Reports Server, Admin Server, and Discover Server. These components usually run as services on the
application tier and may run on one or more nodes. For more details about the components, refer to Oracle
Applications Concepts [B10642-01].
Database Tier
The database tier contains the RDBMS Oracle home, along with the Applications database, which stores all the
data maintained by Oracle Applications. It also contains Oracle Applications processing code stored inside the
Applications database to optimize performance. In essence, the database tier stores the Oracle database files, the
Applications database, and executables. The database does not directly communicate with desktop clients but
rather works with application tier services that mediate the communication between the desktop and database tier.
Top-level Directory Structure
An Oracle EBS Release 12 system utilizes components from many Oracle products. These product files are
stored in a number of key top-level directories on the database and application server machines (Figure 2). These
top-level directories are:
The db/apps_st/data (DATA_TOP) directory is located on the database node machine, and contains
database data files for Oracle Applications.
The db/tech_st/11.1.0 directory is located on the database node machine, and contains Oracle 11g
database software.
The apps/apps_st/appl (APPL_TOP) directory is located on the application node machine, and contains
Oracle Applications product directories such as General Ledger (GL), Purchasing (PO), and many
others.
The apps/apps_st/comn (COMMON_TOP) directory contains common directories and files such as log
files and Java libraries shared across different products.
The apps/tech_st/10.1.2 directory is the top-level technology stack directory for Oracle Developer 10g
(Forms and Reports).
The apps/tech_st/10.1.3 directory is the top-level technology stack directory for the components of
Oracle Application Server 10.1.3. The most significant components used in Oracle Applications R12 are
Oracle HTTP Server (Apache) and Oracle Components for Java (OC4J), which runs most of the code
written in Java.
© 2011 VCE Company, LLC. All rights reserved. 9
Figure 2. Oracle Top-level Directory Structure
EMC Unified Infrastructure Manager 2.1
EMC Ionix
™ Unified Infrastructure Manager (UIM) provides simplified management for Vblock Infrastructure
Platforms, including provisioning, configuration, change, and compliance management. With a consolidated
dashboard view, policy-based management, automated deployment, and deep visibility across the environment,
UIM is integral and essential to effectively and efficiently manage Vblock Infrastructure Platforms.
This solution uses UIM to simplify the tasks required to provision the Vblock infrastructure for an Oracle RAC
implementation. UIM provides the functionality to package portions of the compute, storage, and network
resources of a Vblock infrastructure into infrastructure services. Once in a service, the resources can be reserved,
deployed, allocated, and managed as a single entity through UIM.
© 2011 VCE Company, LLC. All rights reserved. 10
Solution Architecture
VCE solutions are designed to reflect and validate real-world deployments. The solution in this paper describes a
live, non-disruptive migration of the two-node physical Oracle RAC 11g R2 database underlying Oracle EBS to a
virtualized four-node Oracle RAC 11g R2 database. Figure 3 shows the high-level architecture of this solution.
Figure 3. Solution Architecture
© 2011 VCE Company, LLC. All rights reserved. 11
Vblock Architecture
Figure 4 shows the Vblock architecture for this solution.
Figure 4. Vblock Solution Architecture
Validation Environment
The validated solution was built on an Oracle EBS 12.1.1 application configured as follows:
Application Tier: virtualized 2-node application server, FC SAN connected (or NAS NFS over IP)
Database Tier: physical 2-node Oracle RAC 11.2.0.2 database, FC SAN connected
The Oracle RAC nodes communicated with each other through a dedicated 1-Gb private network VLAN, which
also synchronized cache across the various database instances. Two 4-Gb Fibre Channel switches provided FC
SAN connectivity.
© 2011 VCE Company, LLC. All rights reserved. 12
Validation Methodology
Validation of the solution was conducted in several phases, as shown in Figure 5.
Figure 5. Solution Validation Phases
The validation phases consisted of the following phases.
Physical to virtual (P2V)
Phase 1—Create single instance Oracle database on a physical machine.
Phase 2—Convert the Oracle single instance database to an Oracle RAC cluster. Add second physical
node to the cluster.
Phase 3—Add two virtual machines to the cluster.
Phase 4—Add two additional virtual machines to the cluster. Drop the physical nodes from the cluster
(V2P).
Virtual to physical (V2P)
Phase 5—Add two physical machines back into the cluster.
Validation Objectives
The purpose of this validation was to:
Provide customers with a non-disruptive method to migrate their Oracle RAC databases from their
current physical environment to a virtualized environment.
Provide customers with a non-disruptive method to migrate their virtualized (on VMware) Oracle RAC
database environment to a physical environment if Oracle support mandates it.
Demonstrate how customers can reduce planned downtime of Oracle EBS applications using VMware
vMotion to move virtual machines from one physical server to another non-disruptively.
© 2011 VCE Company, LLC. All rights reserved. 13
Hardware Components
A virtual Oracle RAC can be deployed on a Vblock 700, or any Vblock platform that provides SAN connectivity.
The following table details the Vblock 700 hardware components used for the solution.
Component Quantity Configuration
Cisco UCS 1 12 x CPU cores, 192 GB memory
2 x 6120 Fabric Interconnects
4 x Cisco B250-M2 blades (database tier)
2 x B200-M1 or higher performance blades (application tier)
Storage 1 EMC storage
Network 2 Cisco MDS 9222i Switch
Software Components
The following table details the software used for the solution.
Component Quantity Configuration
Oracle E-Business Suite 1 Release 12.1.1
Enterprise Linux (EL) 5 5.4
VMware vSphere 4 4.1
Oracle RAC 1 Version 11.2.0.2
EMC Unified Infrastructure Manager (UIM)
1 2.1
Storage Layout
Raw device mapping (RDM) allows a special file in a VMFS volume to act as a proxy for a raw device. The
mapping file contains metadata used to manage and redirect disk accesses to the physical device. It provides
some of the advantages of a virtual disk in the VMFS file system, while keeping some advantages of direct access
to physical device characteristics. In effect, it merges VMFS manageability with raw device access.
VMware recommends the use of RDM with virtual machine clusters that need to access the same raw LUN for
failover scenarios. The setup is similar to a virtual machine cluster that accesses the same virtual disk file, but an
RDM file replaces the virtual disk file. The VMFS must be configured in shared access mode, to allow more than
one virtual machine to open the mapping file simultaneously.
Figure 6 shows raw device mappings for the Oracle database. Figure 7 shows the layout of Oracle database
storage mapped to the virtualization layer.
© 2011 VCE Company, LLC. All rights reserved. 14
Figure 6. Oracle Database Storage Layout
Figure 7. Mapping Oracle Database Storage to Virtual Layer
© 2011 VCE Company, LLC. All rights reserved. 15
Migrating an Oracle RAC to a Virtual Environment (P2V)
Vblock Infrastructure Platforms allow the live migration of virtual machines from one physical server to another
while continuously powered-up using VMware vMotion™ technology. This process takes place without any
noticeable effect from the point of view of the end user. An administrator can take a virtual machine offline for
maintenance or upgrading without subjecting the system's users to downtime.
One of the most significant advantages of live migration is the fact that it facilitates proactive maintenance. If an
imminent failure is suspected, the potential problem can be resolved before disruption of service occurs. Live
migration can also be used for load balancing, in which work is shared among computers in order to optimize the
utilization of available CPU resources.
The following procedures describe how to create a two-node physical Oracle RAC, migrate the physical Oracle
RAC to a virtual environment, and then release the physical Oracle RAC infrastructure resources for other uses.
Pre-requisites
EMC UIM is used to help simplify the process of creating the Oracle RAC. Before you begin, make sure that UIM:
Is installed according to the instructions in EMC Ionix Unified Infrastructure Manager Version 2.1
Installation Guide.
Is configured with the correct VLANs and Storage Groups according to the instructions in EMC Ionix
Unified Infrastructure Manager Version 2.1 Administration and User Guide.
Has a service offering defined for an Oracle service. Vblock resources are allocated for services
through the service offerings. Service offerings must be made available to the Service Catalog before a
service can be created. Refer to the EMC Ionix Unified Infrastructure Manager Version 2.1
Administration and User Guide for instructions to create service offerings.
Overview of the Steps
Create, provision, and activate the physical Oracle RAC
Create, provision, and activate an ESX server
Migrate the physical Oracle RAC to a virtual Oracle RAC
Remove the physical nodes from the cluster
Deactivate and release the service blades
Phase 1 – Creating the Physical Oracle RAC
This procedure creates, provisions, and activates an Oracle RAC service. Created services are moved to the
Service Manager and are put into a planned state. Services in the planned state can be customized and edited
within the boundaries defined by the service offering constraints. Once planning is complete, the resources of the
service can be provisioned and activated on the Vblock infrastructure.
Step 1: Create the Oracle RAC
1. From the Administration, Service Catalog view, select a Service Offering (in this example, OracleRAC).
2. Click Create Service.
3. Enter the Name and Description of the service. When naming the service ensure that the service name:
© 2011 VCE Company, LLC. All rights reserved. 16
- Is between 2 and 32 characters.
- Does not contain spaces.
- Contains only alphanumeric or the following special characters: decimal point (.), dash (-),
colon (:), underscore (_).
4. Select the specific Vblock platform on which the service will be deployed, and click OK.
5. From the Administration, Service Manager view, select the service to edit (in this example, MyOracleRACService).
6. On the General tab, click Edit.
7. In the Edit General dialog box, select Do not install OS.
Note: UIM does not support automatic installation of Red Hat Linux, so it will be installed manually in a later procedure.
8. On the Servers tab, click Add or Edit.
9. In the Edit Server dialog box:
a. Specify ora-node1 as the first host, and select Gold. Click Save.
b. Specify ora-node2 as the second host, and select Gold. Click Save.
Note: Gold is the pool of blades defined as appropriate for this service offering during the setup phase of UIM.
© 2011 VCE Company, LLC. All rights reserved. 17
10. Leave the Storage layout with the default settings, which were defined in the Service Offering according to Oracle RAC best practices.
11. On the Network tab, click Add or Edit.
12. Define the networking requirements (public network, heartbeat, and Oracle intercommunications) to be bound to the blades.
Note: The network profile is used to set up the connectivity that will be used between the servers of the service and the external networks after the service is activated.
13. Click Save to save the network profile.
© 2011 VCE Company, LLC. All rights reserved. 18
Step 2: Provision the Oracle RAC
Provisioning allocates and reserves the resources, and configures the hardware necessary to realize the service.
Once allocated, the resources are unavailable to other services. For example, the necessary storage is allocated
from the storage pools, the UCS service profiles are created on the UCS, and the blades are reserved. If the
option to install an operating system is selected, the operating system is installed at this time. The provisioning
process in UIM is fully automated.
To provision the Oracle RAC service:
1. From the Administration, Service Manager view, select the service to provision.
2. Click Provision.
3. Follow the progress of the provisioning on the Details, Service History view.
© 2011 VCE Company, LLC. All rights reserved. 19
4. If provisioning is successful, activate the service. If not, troubleshoot the issue. Refer to EMC Ionix Unified Infrastructure Manager Version 2.1 Administration and User Guide for troubleshooting information.
Step 3: Activate the Oracle RAC
Provisioning allocates and reserves the necessary components for the service, as well as installs the operating
system on the blades (if specified during service creation). Activating the service actually turns the system on. The
UCS service profiles are booted on the equipment, which includes activating the network paths and fabric paths.
Note: The operating system will be manually installed and started in a later procedure.
To activate the Oracle RAC service:
1. From the Administration, Service Manager view, select the service to activate.
2. Click Activate. MyOracleRACService has now been provisioned and activated on ora-node1 and ora-node2.
Step 4: Create, provision, and activate the ESX server
ESX/ESXi Server provides the virtualization capabilities that aggregate and present the host hardware to virtual
machines as a normalized set of resources. Use UIM to provision an ESX Server to instantiate the physical
infrastructure for the virtual Oracle RAC nodes.
1. From the Service Catalog, select the ESX server service offering (in the example, vSphereInfrastructure).
2. Click Create Service.
3. Optionally, go to the Service Manager tab to customize the default settings of the service offering.
a. Select the service to customize from the list of services.
b. Click Edit.
c. Make the necessary changes to the Server, Network, and Storage profiles.
4. Go to the Service Manager tab to provision and activate the service.
5. After activating the service, establish connectivity between the ESX servers and the networks selected for service operations.
Note: Make sure you connect the blades with the appropriate networks for public network, heartbeat, and Oracle intercommunications.
© 2011 VCE Company, LLC. All rights reserved. 20
Step 5: Add the ESX server to vCenter
After activating the ESX server service, add it to VMware vCenter using the Connect Host wizard, which prompts
for a host name, user ID, and password. Adding the ESX server to vCenter enables administrators to perform
management tasks including creating, migrating, and cloning virtual machines that reside on the ESX server.
Note: Refer to the VMware vSphere Datacenter Administration Guide for detailed instructions to add a host to vCenter.
© 2011 VCE Company, LLC. All rights reserved. 21
Phase 2 - Creating the Virtual Oracle RAC
Now that you have created the physical Oracle RAC and provisioned an ESX server, create the virtual Oracle
RAC.
Step 1: Create a virtual machine
First, create a single virtual machine to use as a database node, and install the operating system on it. This virtual
machine will be used to create a template from which to clone other virtual machines.
1. Use VMware vSphere to create a new virtual machine (oel5u4-rac1) with the settings shown in the following screen.
2. After creating the virtual machine, install the Enterprise Linux (EL) operating system on it.
Step 2: Create a virtual machine template
For rapid deployment of new virtual machines into the cluster, create a virtual machine template. A template is a
master copy of a virtual machine that you can use to create and provision other virtual machines.
In VMware vSphere, use the Clone Virtual Machine to Template wizard as shown in the following screen. The
template in this document was configured with the requirements and pre-requisites detailed in Table 1.
© 2011 VCE Company, LLC. All rights reserved. 22
Table 1 Virtual Node Template Configuration
Part Description
CPU 4 vCPUs
Memory 20 GB
Operating system Enterprise Linux 5.4
Kernel 2.6.18-164.el5
OS users (users created and passwords set)
Username: grid UserID:1100
Username: oracle UserID:1101
OS groups Group: oinstall GroupID:500 Group: asmadmin GroupID:1100 Group: dba GroupID:400 Group: asmdba GroupID:1200 Group: asmoper GroupID:1201
Software pre-installed Oracle ASMLIB support library
oracleasm-2.6.18-164.el5-2.0.5-1.el5.x86_64.rpm
oracleasmlib-2.0.4-1.el5.x86_64.rpm
oracleasm-support-2.1.4-1.el5.x86_64.rpm
RPM packages installed (as Oracle prerequisites)
See the relevant Oracle installation guide:
Oracle Real Application Clusters Installation Guide 11g
Release 2 (11.2) for Linux
Oracle Grid Infrastructure Installation Guide 11g Release 2
(11.2)
System Configuration (as Oracle prerequisites)
© 2011 VCE Company, LLC. All rights reserved. 23
Step 3: Allocate storage for virtual node
Allocate storage for the virtual node using raw device mapping (RDM). Raw device mapping allows migration of
virtual machines with VMware vMotion. The mapping file acts as a proxy to allow migration of the virtual machine
using the same mechanism that exists for virtual disk files.
Select physical compatibility mode. Physical mode for a raw device mapping specifies minimal SCSI virtualization
of the mapped device, allowing the greatest flexibility for SAN management software. Physical mode also allows
virtual to physical clustering for cost-effective high availability.
1. Use the Hardware tab on the Virtual Machine Properties screen to add raw device mappings (RDMs) for
the virtual node (oel5u4-rac1).
2. Repeat the procedure until all of the LUNs are added to the virtual machine.
Step 4: Install Oracle Clusterware
Oracle RAC enables you to create a cluster of Oracle databases so they operate as a single entity. To create a
cluster, Oracle RAC uses Oracle Clusterware, which is a part of Oracle Grid Infrastructure.
Install the Oracle Grid Infrastructure 11g R2 on the virtual machine. Use the instructions in Oracle Grid
Infrastructure Installation Guide 11g Release 2 for Linux.
© 2011 VCE Company, LLC. All rights reserved. 24
Step 5: Configure SSH for User Equivalence
Before adding the new virtual node to Oracle RAC, set up Secure Shell (SSH) for User Equivalence on the virtual
node for both the grid and oracle users. Refer to Appendix A: Command Examples for an example of how to set
up SSH for user equivalence.
Step 6: Extend Oracle software to the new node
This procedure describes how to add the virtual node to the cluster.
Note: For additional information, refer to Adding and Deleting Oracle RAC Nodes for Oracle E-Business Suite Release 12 [ID 1134753.1] or Oracle Clusterware Administration and Deployment Guide 11g Release 2 (11.2).
1. Before adding the node to the cluster, run the Cluster Verification Utility (CVU) to validate that the new virtual node meets all Oracle prerequisites.
Execute the following command from the primary physical node (oel5u4-node1) as the grid user.
[root@oel5u4-node1 /]# su – grid
[grid@oel5u4-node1 ~]$ cluvfy stage -pre nodeadd -n oel5u4-rac1
2. After successful validation, add the new virtual node (oel5u4-rac1) to the cluster. The following example shows the standard Oracle-supplied script run as the grid user.
[grid@oel5u4-node1 ~]$ cd $Oracle_HOME/oui/bin
[grid@oel5u4-node1 bin]$ ./addNode.sh
"CLUSTER_NEW_NODES={oel5u4rac1}"
"CLUSTER_NEW_PRIVATE_NODE_NAMES={oel5u4-rac1-priv}"
"CLUSTER_NEW_VIRTUAL_HOSTNAMES={oel5u4-rac1-vip}”
3. Run orainstRoot.sh and root.sh on the new node (oel5u4-rac1).
[root@oel5u4-rac1 ~]# /u01/app/oraInventory/orainstRoot.sh
[root@oel5u4-rac1 ~]# /u01/app/11.2.0/grid/root.sh
4. Verify the node has been successfully added. Execute the following command as grid user from the primary node (oel5u4-node1).
[grid@oel5u4-node1 admin]$ cluvfy stage -post nodeadd -n oel5u4-rac1
5. Copy the updated Oracle_HOME from the primary node in the cluster (oel5u4-node1) to the new node.
[root@oel5u4-node1 ~]# su - oracle
[oracle@oel5u4-node1 ~]$ cd $Oracle_HOME/oui/bin
6. Execute the following to add the new node.
a. Run the addNode.sh script on the primary node (oel5u4-node1).
b. Run root.sh and rootcrs.pl on the virtual node (oel5u4-rac1).
[oracle@oel5u4-node1 bin]$ ./addNode.sh "CLUSTER_NEW_NODES={oel5u4-
rac1}"
[root@oel5u4-rac1 ~]#
/u01/app/oracle/product/11.2.0/dbhome_1/root.sh
[root@oel5u4-rac1 ~]# /u01/app/11.2.0/grid/crs/install/rootcrs.pl
© 2011 VCE Company, LLC. All rights reserved. 25
Step 7: Add Oracle RAC instance on new node
After creating a virtual node and adding it to the physical Oracle RAC, use the Database Configuration Assistant
(DBCA) to add a new instance to the new node.
Note: DBCA uses the listener named LISTENER. You can run DBCA on any existing Oracle RAC node but the listener must be enabled and running on that node.
1. Enable the listener on the physical node (oel5u4-node1).
[grid@oel5u4-node1 ~]$ srvctl enable listener -l listener -n oel5u4-
node1
2. Register the instance (VIS1) with the listener on the physical node (oel5u4-node1).
Static registration:
[grid@oel5u4-node1 ~]$ vi
/u01/app/11.2.0/grid/network/admin/listener.ora
Add followings
SID_LIST_LISTENER=
(sid_list=
sid_desc=
(GLOBAL_DBNAME=VIS1)
(Oracle_HOME=/u01/app/oracle/product/11.2.0/dbhome_1)
(SID_NAME=VIS1)
)
)
3. Start the listener on the physical node (oel5u4-node1).
grid@oel5u4-node1 ~]$ srvctl start listener -l listener -n oel5u4-
node1
4. Verify the listener is running.
[grid@oel5u4-node1 ~]$ lsnrctl status listener
5. Launch DBCA to add an instance (VIS3) to the virtual node.
a. Select Oracle Real Application Clusters (RAC) database. Click Next.
b. Select Instance Management. Click Next.
c. Select Add an Instance. Click Next.
d. Click Next to accept the default instance name (VIS3) or change it.
e. Check the summary window.
Note: For additional information, refer to Oracle Real Application Clusters Administration and Deployment Guide 11g Release 2 (11.2).
© 2011 VCE Company, LLC. All rights reserved. 26
Step 8: Register the instance with the listener on the new node
1. Remove the listener entry for the physical node (oel5u4-node1).
[grid@oel5u4-node1 ~]$ vi
/u01/app/11.2.0/grid/network/admin/listener.ora
Delete the followings:
SID_LIST_LISTENER=
(sid_list=
(sid_desc=
(GLOBAL_DBNAME=VIS1)
(Oracle_HOME=/u01/app/oracle/product/11.2.0/dbhome_1)
(SID_NAME=VIS1)
)
)
2. Stop and disable the listener on the physical node (oel5u4-node1).
[grid@oel5u4-node1 ~]$ srvctl stop listener -l listener -n oel5u4-
node1
[grid@oel5u4-node1 ~]$ srvctl disable listener -l listener -n
oel5u4-node1
3. Modify the local_listener parameter on the new virtual node (oel5u4-rac1).
[oracle@oel5u4-rac1 ~]$ sqlplus / as sysdba
SQL> alter system set
local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(H
OST=192.168.104.17)(PORT=1521))))';
4. Register the instance with the listener on the new virtual node.
SQL> alter system register;
System altered.
Step 9: Rebuild context files on database tier
The context file stores all the configuration details of your Oracle Applications system. After you modify this file,
run adconfig.pl to make the configuration changes.
1. Generate a new context file (VIS3_oel5u4-rac1.xml) on the new node.
[oracle@oel5u4-rac1 bin]$ ./adbldxml.pl appsuser=apps
appspasswd=apps
/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/VIS3_oel5u4-
rac1.xml
[oracle@oel5u4-rac1 bin]$
© 2011 VCE Company, LLC. All rights reserved. 27
2. In the new context file, set the value of s_virtual_hostname to point to the hostname for the new virtual
database node.
vi /u01/app/oracle/product/11.2.0/dbhome_1/appsutil/VIS3_oel5u4-
rac1.xml
Changed to : <host oa_var="s_virtual_hostname">oel5u4-rac1-
vip</host>
3. Run adconf.pl to apply the changes to the new virtual database node.
[oracle@oel5u4-rac1 ~]$ cd $Oracle_HOME/appsutil/bin
[oracle@oel5u4-rac1 bin]$ perl adconfig.pl
Enter the full file path to the Context file:
/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/VIS3_oel5u4-
rac1.xml
4. Edit the context file on each physical node in the cluster with the following change to the DB_LISTENER.
[oracle@oel5u4-node1 ~]$ vi $Oracle_HOME/appsutil/VIS1_oel5u4-
node1.xml
<DB_LISTENER oa_var="s_db_listener">VIS</DB_LISTENER>
Changed to:
<DB_LISTENER oa_var="s_db_listener">LISTENER_EBS</DB_LISTENER>
5. Run adautocfg.sh to update the change on these nodes. The example below is for oeL5u4-node1.
[oracle@oel5u4-node1 ~]$ $Oracle_HOME/appsutil/scripts/VIS1_oel5u4-
node1/adautocfg.sh
Step 10: Rebuild context files on application tier
1. Modify the context file on both application nodes (oel5u4-app1 and oel5u4-app2) to include the VIP connection for the new virtual node.
<jdbc_url
oa_var="s_apps_jdbc_connect_descriptor">jdbc:oracle:thin:@(DESCRIPTI
ON=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=
tcp)(HOST=oel5u4-node1-
vip.mordor.vce)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=oel5u4-
node2-
vip.mordor.vce)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=oel5u4-rac1-
vip.mordor.vce)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=VIS)))</jdbc
_url>
2. Run adconfig.sh on each application node to update it with the connection information for the new virtual
node.
[applmgr@oel5u4-app1 ~]$ $AD_TOP/bin/adconfig.sh
contextfile=/u01/local/oracle/VIS/inst/apps/VIS_oel5u4-
app1/appl/admin/VIS_oel5u4-app1.xml
© 2011 VCE Company, LLC. All rights reserved. 28
Note: The following optional step is to make sure that you can connect to either of the application nodes using a Web browser in future.
3. Run adconfig.sh on oel5u4-app1 again.
Step 11: Run Gather Schema Statistics
To ensure the Oracle EBS environment is functioning with no connectivity issues between the application and
database tiers, run a standard application concurrent request, Gather Schema Statistics. For more information
on how to run the concurrent request, refer to the Oracle Applications System Administrator User Guide.
1. Restart the application tier.
2. Run Gather Schema Statistics.
Phase 3 - Creating Additional Virtual Nodes
Use the Deploy Template option in VMware vSphere and the template created earlier (oel5u4-rac1) to create
additional virtual database nodes. In the deployment described in this paper, four virtual database nodes were
created (oel5u4-rac1, oel5u4-rac2, oel5u4-rac3, and oel5u4-rac4).
© 2011 VCE Company, LLC. All rights reserved. 29
Phase 4 - Removing the Physical Nodes from the Cluster
After the Oracle RAC has been migrated to virtual nodes, you can remove the corresponding physical nodes.
After removing the instance and the database software, you deactivate the physical Oracle RAC database
service.
Note: The commands in the following procedures relate to physical node oel5u4-node2. Repeat the procedures for the other physical node: oel5u4-node1.
Step 1: Remove instance and database software from physical node
1. Shut down the application tier.
[applmgr@oel5u4-app1 scripts]$ ./adstpall.sh apps/apps
2. On any virtual node (for example, oel5u4-Rac1), make sure the listener is running on that node.
3. As oracle user, use the Database Configuration Assistant (DBCA) to remove the Oracle RAC instance from the physical node you want to remove (oel5u4-node2).
4. Stop and disable the listener on the virtual node (oel5u4-rac1).
5. Stop and disable the listener on the physical node (oel5u4-node2).
6. Update the inventory on the physical node by running the following commands as oracle user.
[oracle@oel5u4-node2 bin]$ ./runInstaller -updateNodeList
Oracle_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
CLUSTER_NODES={oel5u4-node2} -local
7. Remove Oracle_HOME from the physical node.
[oracle@oel5u4-node2 deinstall]$ ./deinstall -local
8. Update the Oracle RAC inventory.
[oracle@oel5u4-rac1 bin]$ ./runInstaller -
updateNodeListOracle_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
"CLUSTER_NODES={oel5u4-rac1,oel5u4-rac2,oel5u4-rac3,oel5u4-rac4}"
9. Verify the node has been removed from inventory.xml.
[oracle@oel5u4-rac4 ~]$ vi /u01/app/oraInventory/ContentsXML/inventory.xml
Step 2: Remove Clusterware from the physical node
To remove the Oracle Clusterware, execute the following:
[root@oel5u4-node2 ~]# cd /u01/app/11.2.0/grid/crs/install
[root@oel5u4-node2 install]# ./rootcrs.pl -deconfig -force
© 2011 VCE Company, LLC. All rights reserved. 30
Step 3: Delete the physical node
From one of the virtual nodes (for example, oel5u4-rac1), remove the physical node from the cluster.
1. As root user, remove the physical node (oel5u4-node2) from the cluster.
[root@oel5u4-rac1 ~]# cd /u01/app/11.2.0/grid/bin
[root@oel5u4-rac1 bin]# ./crsctl delete node -n oel5u4-node2
2. On the physical node you are removing, run the following to update the inventory.
[grid@oel5u4-node2 bin]$ ./runInstaller -updateNodeList
Oracle_HOME=/u01/app/11.2.0/grid "CLUSTER_NODES={oel5u4-node2}"
CRS=TRUE -silent -local
3. As grid user, de-install the Oracle Clusterware on the physical node you are deleting.
[grid@oel5u4-node2 ~]$ /u01/app/11.2.0/grid/deinstall/deinstall –
local
4. Run the following commands as the root user or the administrator on the physical node (oel5u4-node2). Press Enter after you finish running the commands.
Note: The deconfig commands can be executed in parallel on all remote nodes. Execute the command on the local node after execution completes on all the remote nodes.
/tmp/deinstall2011-03-03_07-41-06PM/perl/bin/perl
-I/tmp/deinstall2011-03-03_07-41-06PM/perl/lib
-I/tmp/deinstall2011-03-03_07-41-06PM/crs/install
/tmp/deinstall2011-03-03_07-41-06PM/crs/install/rootcrs.pl -force -
deconfig –paramfile
"/tmp/deinstall2011-03-03_07-41-06PM/response
deinstall_Ora11g_gridinfrahome1.rsp"
5. Run the following to update the inventory.
[grid@oel5u4-rac1 bin]$ ./runInstaller -updateNodeList
Oracle_HOME=/u01/app/11.2.0/grid "CLUSTER_NODES={oel5u4-rac1,oel5u4-
rac2,oel5u4-rac3,oel5u4-rac4}" CRS=TRUE –silent
6. Verify the physical node was deleted.
[grid@oel5u4-rac1 bin]$ cluvfy stage -post nodedel -n oel5u4-node
Step 4: Perform post-removal tasks
1. Update the application tier and database tier.
[oracle@oel5u4-rac1 ~]$ sqlplus apps/apps
SQL> exec fnd_conc_clone.setup_clean;
2. Reconfigure the context on every existing database node. For example:
© 2011 VCE Company, LLC. All rights reserved. 31
[oracle@oel5u4-node1 ~]$ $Oracle_HOME/appsutil/scripts/VIS1_oel5u4-
node1/adautocfg.sh
3. Edit the context file on the two application nodes to remove the deleted physical node.
4. Run adconfig.sh on both application nodes.
[applmgr@oel5u4-app1 ~]$ $AD_TOP/bin/adconfig.sh
contextfile=/u01/local/oracle/VIS/inst/apps/VIS_oel5u4-
app1/appl/admin/VIS_oel5u4-app1.xml
[applmgr@oel5u4-app2 ~]$ $AD_TOP/bin/adconfig.sh
contextfile=/u01/local/oracle/VIS/inst/apps/VIS_oel5u4-
app2/appl/admin/VIS_oel5u4-app2.xml
5. Run autoconfig.sh on oel5u4-app1 again.
[applmgr@oel5u4-app1 ~]$ $AD_TOP/bin/adconfig.sh
contextfile=/u01/local/oracle/VIS/inst/apps/VIS_oel5u4-
app1/appl/admin/VIS_oel5u4-app1.xml
6. Restart the application tier.
[applmgr@oel5u4-app1 admin]$ cd $ADMIN_SCRIPTS_HOME
[applmgr@oel5u4-app1 scripts]$ ./adstrtal.sh apps/apps
7. To ensure the Oracle EBS environment is functioning with no connectivity issues between the application and database tiers, run a standard application concurrent request, Gather Schema Statistics. For more
information on how to run the concurrent request, refer to the Oracle Applications System Administrator User Guide.
© 2011 VCE Company, LLC. All rights reserved. 32
Step 5: Power down the server and disassociate the service profile
1. Login to UCS Manager.
2. In the navigation pane, click the Servers tab.
3. Select the service profile for the blade.
4. Select KVM Console from the right-click menu.
5. In the console, click Shutdown Server.
8. Click OK.
9. Click OK again to shut down the server. The server begins shutting down.
6. On the FSM tab, monitor status of blade shutdown to make sure it finishes.
© 2011 VCE Company, LLC. All rights reserved. 33
Step 6: Deactivate and release the service blades
Deactivating a service shuts down the UCS service profiles while preserving the service configuration and
resources. Once a service is inactive, the blades can be released for use in another service or for an application
outside of UIM.
During deactivation, the service profiles are booted down. All of the configuration and resources are preserved,
identities such as MAC addresses and World Wide Port Names (WWPNs) are kept in tact, and the storage is
preserved on the storage array. Optionally, the blades can be released after the service is deactivated. The
storage, however, must stay reserved to ensure the data is preserved for reactivation.
1. From the Administration, Service Manager view, select the active service to deactivate (in the example, MyOracleRACService).
2. Click Deactivate.
3. Wait for deactivation to complete, which is indicated by an Inactive status.
4. Click Release Blades.
© 2011 VCE Company, LLC. All rights reserved. 34
Phase 5 – Adding a Physical Node to the Oracle RAC Cluster (V2P)
The following sections describe how to add a physical node (oel5u4-node2) to the virtualized Oracle RAC. This
operation enables you to meet any support requirement by Oracle to re-produce problems in a physical
environment.
To add a physical server to the cluster, use UIM to re-activate the physical Oracle RAC service, and then add a
physical node to the Oracle RAC. Re-activating the service turns the system on. The UCS service profiles are
booted on the equipment, which includes re-activating the network paths and fabric paths.
Step 1: Re-activate the Oracle RAC service
1. Confirm that the Vblock setup will still support the service.
a. Go to the Administration, Vblock Platforms tab.
b. Locate the Vblock platform on which you are reactivating the service.
c. Check the alert and status messages in the first column of the row and respond as indicated in the UIM online help.
2. Validate that the service is still in compliance.
a. From the Administration, Service Manager view, select the inactive service to activate.
b. Click Verify Compliance.
c. Once the test is finished, check the Alert column and respond as indicated in the UIM online help.
3. From the Administration, Service Manager view, select the service to activate (in the example, MyOracleRACService).
4. Click Activate.
© 2011 VCE Company, LLC. All rights reserved. 35
Step 2: Add the physical Oracle RAC node
1. Configure SSH for User Equivalence for the node you are adding. Refer to Appendix A: Command Examples for an example of configuring SSH for User Equivalence.
2. Execute the following to verify the node does not already exist.
[grid@oel5u4-rac1 ~]$ cluvfy stage -pre nodeadd -n oel5u4-node2
3. Extend Oracle Clusterware to the new node.
Note: Refer to the checks and steps in Oracle Clusterware Administration and Deployment Guide 11g Release 2 (11.2).
[grid@oel5u4-node1 bin]$ ./addNode.sh
"CLUSTER_NEW_NODES={oel5u4rac1}"
"CLUSTER_NEW_PRIVATE_NODE_NAMES={oel5u4-rac1-priv}"
"CLUSTER_NEW_VIRTUAL_HOSTNAMES={oel5u4-rac1-vip}”
4. When the script has completed successfully and script output has been validated, run root.sh on the new
node as root user.
[root@oel5u4-node2 ~]# /u01/app/11.2.0/grid/root.sh
5. Extend the Oracle software to new node.
Execute the following from $Oracle_HOME/oui/bin on an existing node in the cluster:
[root@oel5u4-rac1 ~]# su – oracle [oracle@oel5u4-rac1 ~]$ cd $Oracle_HOME/oui/bin [oracle@oel5u4-rac1 bin]$ ./addNode.sh "CLUSTER_NEW_NODES={oel5u4-node2}"
6. Run root.sh on the new node:
[root@oel5u4-node2 ~]#
/u01/app/oracle/product/11.2.0/dbhome_1/root.sh
7. Optionally, execute rootcrs.pl to check the CRS (Cluster Ready Services) resource for errors.
[root@oel5u4-node2 ~]# /u01/app/11.2.0/grid/crs/install/rootcrs.pl
8. Add an Oracle RAC database instance to the new node.
Note: Refer to Oracle Real Application Clusters Administration and Deployment Guide 11g Release 2 (11.2) for detailed instructions.
a. Make sure the listener is running on any existing database node (for example, oel5u4-rac1).
b. As oracle user on any existing database node, run the Database Configuration Assistant (DBCA) to add an Oracle RAC database instance (VIS7).
9. Rebuild the context on the database tier.
a. Generate a context file on the new physical node (oel5u4-node2).
[oracle@oel5u4-node2 .ssh]$ cd $Oracle_HOME/appsutil/bin
b. Edit the context file to the following to change the hostname of the new node to its VIP hostname.
© 2011 VCE Company, LLC. All rights reserved. 36
<jdbc_url
oa_var="s_apps_jdbc_connect_descriptor">jdbc:oracle:thin:@(DESCRIPTI
ON=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=
tcp)(HOST=oel5u4-rac4-
vip.mordor.vce)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=oel5u4-rac1-
vip.mordor.vce)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=oel5u4-rac2-
vip.mordor.vce)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=oel5u4-rac3-
vip.mordor.vce)(PORT=1521))(ADDRESS=(PROTOCOL=tcp)(HOST=oel5u4-
node2-
vip.mordor.vce)(PORT=1521)))(CONNECT_DATA=(SERVICE_NAME=VIS)))</jdbc
_url
c. On the newly added physical node, alter the local_listener parameter to make it aware of the new
database instance (VIS7).
[oracle@oel5u4-rac1 ~]$ sqlplus / as sysdba
SQL> alter system set
local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(H
OST=192.168.104.17)(PORT=1521))))';
d. Run adconfig.sh on all physical database nodes. For example:
[oracle@oel5u4-node2 bin]$ cd $Oracle_HOME/appsutil/bin
[oracle@oel5u4-node2 bin]$ perl adconfig.pl
Enter the full file path to the Context file:
/u01/app/oracle/product/11.2.0/dbhome_1/appsutil/VIS7_oel5u4-
node2.xml
10. Run autoconfig.sh on each application nodes (oel5u4-app1 and oel5u4-app2).
11. Run autoconfig.sh on oel5u4-app1 again.
12. Start the application tier.
13. To ensure the Oracle EBS environment is functioning with no connectivity issues between the application and database tiers, run a standard application concurrent request, Gather Schema Statistics. For more information on how to run the concurrent request, refer to the Oracle Applications System Administrator User Guide.
© 2011 VCE Company, LLC. All rights reserved. 37
Using VMware vMotion to Reduce Planned Downtime
A typical Oracle EBS solution consists of multiple layers and instances running simultaneously and working
together to provide failover and load balancing. VMware vMotion enhances this capability by enabling the live
migration of running virtual machines from one physical server to another with zero downtime, continuous service
availability, and complete transaction integrity. Live migration of virtual machines allows hardware maintenance to
be performed without scheduling downtime and disrupting business operations.
VMware vMotion is a key enabling technology for creating the dynamic, automated, and self-optimizing data
center. This application makes it possible to service hardware and software with no disruption so that you can
configure, manage, maintain, and update your Oracle EBS applications without the risk of the application going
offline.
For maintaining high availability, vMotion enables you to:
Perform hardware maintenance without scheduled downtime
Proactively migrate virtual machines away from failing or underperforming servers
Keep your resources optimized to align with business priorities
The migration of a virtual machine with vMotion preserves the precise execution state, the network identity, and
the active network connections. As a result, there is zero downtime and no disruption to the user. This ensures
that your Oracle EBS applications remain online, even under conditions in which a non-virtualized instance of the
application becomes unavailable.
© 2011 VCE Company, LLC. All rights reserved. 38
Performing Live Migration
In this example, the virtual machine (oel5u4-app1) is an Oracle EBS application node.
To migrate a virtual machine to a different physical host:
1. Login to the VMware vSphere Client.
2. In the navigation pane, right-click the virtual machine (oel5u4-app1), and then select Migrate from the menu.
3. Select Change Host. Click Next.
© 2011 VCE Company, LLC. All rights reserved. 39
4. Select the destination host or cluster (192.168.50.94). Click Next.
5. For better performance, select High Priority for vMotion Priority. Click Next.
© 2011 VCE Company, LLC. All rights reserved. 40
6. Click Finish to start the migration.
Verifying the Live Migration is Non-disruptive
To verify live migration is non-disruptive:
1. Login to Oracle EBS.
2. Run the Concurrent program, Gather Schema Statistics.
© 2011 VCE Company, LLC. All rights reserved. 41
3. In vSphere Client, migrate the virtual machine (oel5u4-App1) back to the original host (192.168.50.95) while Gather Schema Statistics is running.
a. In the navigation pane, right-click the virtual machine (oel5u4-App1), and then select Migrate from
the menu.
b. Select Change Host. Click Next.
c. Select the destination host or cluster (192.168.50.95). Click Next.
d. Select High Priority. Click Next.
e. Click Finish to start the migration.
© 2011 VCE Company, LLC. All rights reserved. 42
Scalability and Performance Considerations for Oracle EBS and RAC
There are several key factors to consider when sizing Oracle E-Business Suite systems for RAC environments on
the Vblock infrastructure. These factors include:
Virtual machine configuration (vCPUs, memory, network configuration)
Network configuration with Cisco switches
Storage performance and sizing for the EMC VMAX™
Oracle Release12 EBS configuration
Virtual machine configuration can take advantage of the VMware Distributed Resource Scheduler (DRS) load
distribution technology to balance the workload of each virtual machine in the Vblock architecture used to host the
Oracle EBS application.
VMware Distributed Resource Scheduler (DRS) continuously monitors utilization across a resource pool and
intelligently allocates available resources among virtual machines according to business needs. When a virtual
machine experiences increased load, VMware DRS first evaluates its priority against established resource
allocation rules and policies, and if justified, allocates additional resources. Resources are allocated to the virtual
machine by either migrating it to another server with more available resources or by making more ―space‖ for it on
the same server by migrating other virtual machines to different servers. The live migration of virtual machines to
different physical servers is executed completely transparent to end-users through VMware vMotion.
Network performance is an important consideration when sizing the performance of Oracle EBS applications. By
tuning the Cisco Nexus switches deployed within the Vblock infrastructure, you can optimize the available network
traffic sent between all of the nodes within the application, concurrent managers, and database tiers for Oracle
EBS. In addition, to enhance performance and to scale out, either software-based load balancing can be deployed
for Oracle or hardware-based load balancers such as Cisco ACE can be implemented to take advantage of
enhanced performance and security.
The storage array used with the Vblock platform should be tuned based on best practices available from Oracle
and EMC. In addition, tuning the Oracle application tier servers and database server will allow further performance
and scalability enhancements.
© 2011 VCE Company, LLC. All rights reserved. 43
Conclusion
With Vblock™ Infrastructure Platforms, VCE delivers the industry's first completely integrated IT offering that
combines best-in-class virtualization, networking, computing, storage, security, and management technologies
with end-to-end vendor accountability.
This paper was written to demonstrate the simplicity and ease with which you can virtualize a physical Oracle
RAC database on the Vblock platform. Virtualizing Oracle EBS applications on the Vblock platform provides a
number of key values:
Containment of server sprawl and increased server utilization
Dynamic, non-disruptive migration of Oracle instances on virtual machines to other systems to perform
hardware maintenance or system changes without disruption
Improved operational agility through live virtual machine migration to accommodate dynamic workloads
Fast provisioning of pre-configured Oracle EBS environments using vSphere templates
Automatic detection and restart of failed Oracle EBS virtual machines with VMware HA
© 2011 VCE Company, LLC. All rights reserved. 44
Appendix A: Configuring SSH for User Equivalence
Configuring SSH for User Equivalence provides a secure means of copying files and executing programs on
nodes in the cluster without requiring passwords prompts. Secure Shell configuration is covered in detail in both
the Oracle Real Application Clusters Installation Guide and Oracle Grid Infrastructure Installation Guide.
Note: Refer to Appendix E in Oracle Grid Infrastructure Installation Guide 11g Release 2 (11.2) for further details on manual set up of SSH.
The main steps to configure SSH for User Equivalence are:
1. Create authentication keys (rsa and dsa) using ssh-keygen on the new virtual node (oel5u4-rac1) as the grid
user.
2. Authorize access to the new virtual node using the local key by copying the public key into the ~/.ssh/authorized_keys file (for example, [grid@oel5u4-rac1 .ssh]$ cat id_rsa.pub >>
authorized_keys)
3. Copy the ~/.ssh/authorized_keys files to other systems to allow access from the local system (for example, [grid@oel5u4-node1 .ssh]$ scp authorized_keys oel5u4-
rac1:/home/grid/.ssh)
4. Repeat this procedure for each newly added virtual node and other existing physical nodes in the cluster.
Figure 8 is an example of configuring SSH for User Equivalence.
© 2011 VCE Company, LLC. All rights reserved. 45
[root@oel5u4-rac1 ~]# su – grid
[grid@oel5u4-rac1 ~]$ /usr/bin/ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_rsa):
Created directory '/home/grid/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_rsa.
Your public key has been saved in /home/grid/.ssh/id_rsa.pub.
The key fingerprint is:
f0:97:07:16:37:38:29:3f:d0:b7:9f:23:9f:d9:41:27 grid@oel5u4-
rac1.mordor.vce[grid@oel5u4-rac1 ~]$ /usr/bin/ssh-keygen -t dsa
Generating public/private dsa key pair.
Enter file in which to save the key (/home/grid/.ssh/id_dsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/grid/.ssh/id_dsa.
Your public key has been saved in /home/grid/.ssh/id_dsa.pub.
The key fingerprint is:
f9:0b:06:c4:44:2f:cf:2a:0c:eb:68:03:cd:be:87:16 grid@oel5u4-
rac1.mordor.vce
[grid@oel5u4-rac1 ~]$ cd .ssh
[grid@oel5u4-rac1 .ssh]$ cat id_rsa.pub >> authorized_keys
[grid@oel5u4-rac1 .ssh]$ cat id_dsa.pub >> authorized_keys
[grid@oel5u4-node1 ~]$ cd .ssh
[grid@oel5u4-node1 .ssh]$ ssh oel5u4-rac1 cat
/home/grid/.ssh/authorized_keys >> authorized_keys
[grid@oel5u4-node2 .ssh]$ ssh oel5u4-rac1 cat
/home/grid/.ssh/authorized_keys >> authorized_keys
[grid@oel5u4-node1 .ssh]$ scp authorized_keys oel5u4-
rac1:/home/grid/.ssh
Figure 8. Example of Configuring SSH for User Equivalence
© 2011 VCE Company, LLC. All rights reserved. 46
ABOUT VCE
VCE, the Virtual Computing Environment Company formed by Cisco and EMC with investments from VMware and Intel,
accelerates the adoption of converged infrastructure and cloud-based computing models that dramatically reduce the
cost of IT while improving time to market for our customers. VCE, through the Vblock platform, delivers the industry's first
completely integrated IT offering with end-to-end vendor accountability. VCE's prepackaged solutions are available
through an extensive partner network, and cover horizontal applications, vertical industry offerings, and application
development environments, allowing customers to focus on business innovation instead of integrating, validating and
managing IT infrastructure.
For more information, go to www.vce.com.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS IS." VCE MAKES NO REPRESENTATIONS
OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND
SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OR MECHANTABILITY OR FITNESS FOR A
PARTICULAR PURPOSE.
Copyright © 2011 VCE Company, LLC. All rights reserved. Vblock and the VCE logo are registered trademarks or trademarks of VCE Company, LLC. and/or its affiliates in the United States or other countries. All other trademarks used herein are the property of their respective owners.