white paper dell reference configuration deploying oracle...

White Paper

Dell Reference Configuration

Deploying Oracle® Enterprise Manager 10g Grid Control R3 on Dell™ PowerEdge™ Servers,

Dell|EMC™ and Dell PowerVault™ Storage

Abstract This white paper provides an architectural overview and configuration guidelines for deploying Oracle Enterprise Manager Grid Control 10g R3 on Dell PowerEdge servers. The white paper illustrates high availability deployment of the Oracle Management Repository with Dell|EMC and Dell PowerVault storage. Using the knowledge gained through joint development, testing and support with Oracle, this Dell Reference Configuration documents “best practices” that can help speed Oracle solution implementation and help simplify operations, improve performance and availability. July, 2007

Dell Reference Configuration for Oracle Enterprise Manager Grid Control 1


THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. . © 2007 Dell Inc. All rights reserved. Reproduction in any manner whatsoever without the written permission of Dell Inc. is strictly forbidden. Trademarks used in this text: Dell, the DELL logo, PowerEdge and PowerVault are trademarks of Dell Inc.; Intel and Xeon are registered trademarks of Intel Corporation; EMC, Navisphere, and PowerPath are registered trademarks of EMC Corporation; Microsoft, Windows, and Windows Server are registered trademarks of Microsoft Corporation; Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell Inc. disclaims any proprietary interest in trademarks and trade names other than its own. July 2007 Rev. A01


Table of Contents ABSTRACT .................................................................................................................................................. 1 INTRODUCTION ........................................................................................................................................ 4

DELL SOLUTIONS FOR ORACLE DATABASE 10G ...................................................................................... 4 OVERVIEW OF THIS WHITE PAPER .......................................................................................................... 4

ARCHITECTURE OVERVIEW - DELL SOLUTION FOR ORACLE ENTERPRISE MANAGER 10G GRID CONTROL R3........................................................................................................................... 5 COMPATIBILITY MATRIX FOR OEM GRID CONFIGURATIONS ................................................ 7 DELL OPENMANAGE OVERVIEW ....................................................................................................... 7 INTEGRATION OF DELL OPENMANAGE WITH ORACLE ENTERPRISE MANAGER GRID CONTROL.................................................................................................................................................... 8 HARDWARE CONFIGURATION - DELL SOLUTION FOR ORACLE ENTERPRISE MANAGER 10G GRID CONTROL R3........................................................................................................................... 8

STORAGE CONFIGURATION FOR THE ORACLE MANAGEMENT REPOSITORY (OMR)................................... 8 Configuring OMR on Dell PowerVault MD3000 Storage with Dual SAS 5/E Controller Connections8 Configuring OMR on Dell|EMC Fibre Channel Storage with Dual HBAs and Dual Fibre Channel Switch Connections............................................................................................................................... 9 Configuring Virtual Disks / LUNs ...................................................................................................... 10

SERVER CONFIGURATION FOR THE ORACLE MANAGEMENT REPOSITORY................................................ 13 Configuring Fully Redundant Ethernet Interconnects ........................................................................ 13 Configuring Dual SAS 5/E Controllers for MD3000 Storage............................................................. 14 Configuring Dual HBAs for Dell|EMC storage.................................................................................. 14 Configuring Dual NICs for Private Network ...................................................................................... 14

SERVER CONFIGURATION FOR THE ORACLE MANAGEMENT SERVICE (OMS) .......................................... 14 Configuring Network requirement ...................................................................................................... 14

SOFTWARE CONFIGURATION - DELL SOLUTION FOR ORACLE ENTERPRISE MANAGER 10G GRID CONTROL R3......................................................................................................................... 15

OPERATING SYSTEM CONFIGURATION ..................................................................................................... 15 Configuring File System Partitions on Red Hat Enterprise Linux 4.4................................................ 15 Configuring the Private NIC Teaming................................................................................................ 15 Configuring the Same Public Network Interface Name ...................................................................... 15 Configuring SSH and RSH.................................................................................................................. 15 Configuring Shared Storage for the Database using the ASM Library Driver................................... 16 Installing Dell OpenManage Server Administration (OMSA) 5.2 on Red Hat Enterprise Linux 4.4 x86_64................................................................................................................................................. 16

ORACLE ENTERPRISE MANAGER 10G GRID CONTROL R3 CONFIGURATION............................................. 16 Configuring the RAC database that will support the OMR................................................................. 16 Configuring DBMS_SHARED_POOL package for OMR................................................................... 17 Configuring Database initialization parameters for the OMR ........................................................... 17 Configuring the Oracle Management Service (OMS)......................................................................... 17 Specifying Repository Database Configuration during OMS installation .......................................... 18 Configuring Additional Oracle Management Agents (OMA) ............................................................. 18 Configuring Integration of Oracle Enterprise Manager 10g Grid Control R3 with OMSA 5.2 ......... 18

CONFIGURATION DELIVERABLES LIST - DELL SOLUTION FOR ORACLE ENTERPRISE MANAGER 10G GRID CONTROL R3 ON RED HAT ENTERPRISE LINUX 4.............................. 19 CONCLUSION........................................................................................................................................... 22 REFERENCES ........................................................................................................................................... 23


Introduction Dell™ PowerEdge™ servers, Dell PowerVault™ , and Dell|EMC™ storage systems are ideal choices to deploy highly reliable and sustainable Oracle 10g databases and Oracle Enterprise Manager Grid Control. This Reference Configuration white paper is intended to help IT professionals design and configure Oracle 10g Enterprise Manager Grid Control solutions using Dell servers and storage that apply “best practices” derived from laboratory and real-world experiences. This white paper documents Dell’s recommended approach for implementing a tested and validated solution for Oracle 10g R3 Enterprise Manager Grid Control on Dell PowerEdge 9th generation servers, PowerVault MD3000, Dell|EMC CX3-20 storage arrays with Red Hat Enterprise Linux 4 (RHEL4).

Dell Solutions for Oracle Database 10g Dell Solutions for Oracle Database 10g are designed to simplify operations, improve utilization and cost-effectively scale as your needs grow over time. In addition to providing price/performance leading server and storage hardware, Dell Solutions for Oracle 10g include:

• Dell Configurations for Oracle – in-depth testing of Oracle 10g configurations for the most in-demand solutions; documentation and tools that help simplify deployment

• Integrated Solution Management – standards-based management of Dell Solutions for Oracle 10g that lower operational costs through integrated hardware and software deployment, monitoring and update

• Oracle Licensing multiple licensing options that can simplify customer purchase • Dell Enterprise Support and Professional Services for Oracle – offerings for the planning,

deployment and maintenance of Dell Solutions for Oracle 10g For more information concerning Dell Solutions for Oracle Database 10g, please visit www.dell.com/oracle. Overview of this White Paper The balance of this white paper will provide the reader with a detailed view of the Dell Reference Configuration for deploying Oracle 10g R3 Enterprise Manager Grid Control on RHEL 4.4, best practices for configuring the hardware and software components and pointers for obtaining more information.

http://www.dell.com/oracle

Architecture Overview - Dell Solution for Oracle Enterprise Manager 10g Grid Control R3 The Dell Reference Configuration for Oracle Enterprise Manager 10g Grid Control R3 is intended to validate the Grid Control architecture which consists of the following components:

• Oracle Management Agent (OMA) running on Dell PowerEdge monitored servers • Oracle Management Service (OMS) running on a Dell PowerEdge server running RHEL4.4 • Oracle Management Repository (OMR) RAC database instances running on Dell PowerEdge

servers • Server-storage interconnect • Dell|EMC FC Storage and Dell PowerVault MD3000 SAS storage system with MD1000

expansion This reference configuration supports the Oracle Maximum Availability Architecture (MAA) by placing the OMR tier in an Oracle 10g RAC database as part of the Oracle Enterprise Manager Grid Control Framework. Moreover, the OMR database resides on a separate host in this architecture and does not compete for resources with the OMS. This approach can enhance the availability, scalability and flexibility of the management system. Dell has validated two configurations with the Oracle Enterprise Manager 10g Grid Control R3 on 64-bit RHEL4.4. An architectural overview of the first configuration is shown in Figure 1 below. The architecture is made of the following components: • Single node Dell PowerEdge 2950 server where OMS is running • Two node Dell PowerEdge 2970 server cluster hosting the OMR RAC database • Server-storage interconnect using Serial Attached SCSI (SAS) • Dell PowerVault storage MD3000 with MD1000 expansion on which the OMR database resides,

supporting up to 45 drives

Figure 1 - Architectural Overview of Enterprise Manager Grid Control Direct-Attached SAS

Storage Configuration


An architectural overview of the second configuration is shown in Figure 2 below. This architecture is made of the following components:

• Single node Dell PowerEdge 1950 server where OMS is running • Two node Dell PowerEdge 2950 server cluster hosting the OMR RAC database • 4G Fibre Channel switches for a SAN environment • Dell|EMC CX3-20 Fibre Channel storage system on which the OMR database resides

Figure 2 - Architectural Overview of Enterprise Manager Grid Control Fibre Channel Storage Configuration

Dell PowerEdge servers are designed to deliver the highest performance for critical enterprise applications like database, messaging, web services and infrastructure applications. As proprietary systems are increasingly replaced by industry-standard systems, applications like databases, high performance computing clusters and messaging systems can take advantage of the performance and scalability of the PowerEdge servers. Combined with Dell storage systems, customers can easily deploy these PowerEdge servers as building blocks of a scalable enterprise, consolidating and virtualizing both the computing resources as well as the storage resources. The PowerVault MD3000 is a high-performance storage array built for critical applications running on one, two or a pair of clustered PowerEdge servers. Additionally, the storage array can support up to four servers in environments where redundant host connections are not required. The PowerVault MD3000 includes dual active/active RAID controllers with mirrored cache and multi-path I/O management which helps ensure that storage processing continues without disruption. Other high-availability features include hot-pluggable, redundant power supplies, cooling modules and disk drives, active disk scrubbing and non-disruptive firmware upgrades. The Dell|EMC storage subsystem delivers advanced storage capabilities that are not found in other midrange storage systems, including simple management tools, continuous data availability and integrity, industry-leading price/performance, data mobility, and scalability between multiple storage tiers. The Dell|EMC storage subsystem is offered in various models, ranging from affordable entry-level solutions to high-performance, maximum-capacity configurations for your most demanding requirements. All Dell|EMC CX3 series arrays support advanced software including local replication for backup/restore,



remote replication for disaster recovery and data mobility. The Dell|EMC is architected with two storage processors to guard against a single point of failure.

Compatibility Matrix for OEM Grid Configurations We can have several combinations of OMS, OMA and OMR for the deployment of OEM Grid control framework. The table 1 provides the matrix for OEM Grid Control deployment.

Oracle Version (Target Database)

Oracle Management Service (OMS) Version

Oracle Management Agent (OMA) Version

Oracle Management Repository (OMR) Version

9.2.0.6 + 10.1.0.4 + 10.2.0.2 +

10.2.0.3 10.2.0.3 9.2.0.6 + 10.1.0.4 + 10.2.0.2 +

9.2.0.6 + 10.1.0.4 + 10.2.0.2 +

10.2.0.2 10.2.0.2 9.2.0.6 + 10.1.0.4 + 10.2.0.2 +

9.2.0.4 10.1.0.4 + 10.2.0.2

10.1.0.5 10.1.0.5 9.2.0.4 10.1.0.4 + 10.2.0.2

Table 1 - Compatibility Matrix for OEM Grid Control Framework

Though one can have several combinations of OMS, OMA and OMR this reference configuration has been deployed with the versions as provided the table 2 below.

Oracle Version (Target Database)

Oracle Management Service (OMS) Version

Oracle Management Agent (OMA) Version

Oracle Management Repository (OMR) Version

10.2.0.3 10.2.0.3 10.2.0.3 10.2.0.3

Table 2 - Reference Configuration for OEM Grid Control Framework

Dell OpenManage Overview

Dell's approach to systems management is to provide inherently manageable, standards-based platforms along with a comprehensive set of standards-based tools for proactive management throughout the computing life cycle. Dell server, storage, networking, and client solutions are designed to help simplify and automate the administration of your technology resources-and to help you control your IT investment.

The advantages of our interoperable management solutions derive from Dell's commitment to:

• Open manageability —Dell's instrumented clients, servers, storage, printers and network platforms interface seamlessly with most standards-based management tools and consoles. Dell systems management solutions and platforms provide the pro-active management information and control functions you need to optimize deployment, health status monitoring, fault recovery, change management and more.

• Industry standards —Dell champions open standards within the industry because they are the foundation for management systems that can deploy, monitor and upgrade heterogeneous computing environments. Standards also give you greater choice in the selection of your systems management solution, providing more flexibility to better meet your specific requirements.


• Strong partnerships —Dell partners with industry-leading companies to deliver integrated, "best-in-class" technologies, services and standardized components to provide customers with cost-effective broad-based systems management functionality.

Integration of Dell OpenManage with Oracle Enterprise Manager Grid Control Oracle Enterprise Manager 10g Grid Control offers monitoring of Dell PowerEdge servers via integration with Dell OpenManage software. Key functions enabled by this integration include

o Discovery of PowerEdge servers on which OpenManage has been deployed o Inventory reporting of PowerEdge server hardware configuration o Monitoring of PowerEdge server hardware health including alerting to potential or actual faults of

disk drives, power supplies, fans and more o Event driven policy execution based on reported PowerEdge hardware faults

For details on installing Dell OpenManage software and integrating with Oracle Enterprise Manager, please refer to the Software Configuration section.

Hardware Configuration - Dell Solution for Oracle Enterprise Manager 10g Grid Control R3

Storage Configuration for the Oracle Management Repository (OMR) The Oracle Management Repository is hosted in an Oracle RAC database in a high-availability configuration. The following sections describe the storage configuration necessary to setup the OMR on a Dell PowerVault MD3000 storage system or a Dell | EMC storage system.

Configuring OMR on Dell PowerVault MD3000 Storage with Dual SAS 5/E Controller Connections Figure 3 illustrates the interconnection of the two-node PowerEdge cluster hosting the OMR on an MD3000 storage system with two MD1000 expansion units. Using dual SAS 5/E HBAs in each server with an Active/Active configuration, each server has a redundant pathway to the controllers and the virtual disks. If either a SAS 5/E HBA or an MD3000 RAID controller module fails, operations continue using the remaining SAS 5/E – MD3000 storage controller pair. The MD3000 can be daisy-chained with up to two MD1000 expansion units, providing access to a maximum of 45 disks in the entire storage array. Figure 3 illustrates the cabling method in which two MD1000 enclosures can be cascaded from an MD3000 enclosure.

Figure 3 - Cabling a Direct Attached MD3000 Storage Array with two MD1000 expansion units

The physical disks in the MD3000 storage array provide the physical storage capacity for the Oracle RAC database on which the OMR will be stored. Before data can be stored, the MD3000 physical storage capacity must be configured into components, known as disk groups and virtual disks. A disk group is a set of physical disks that are logically grouped and assigned a RAID level. Each disk group created provides the overall capacity needed to create one or more virtual disks, which are logical entities that the server uses to store data. For details on virtual disk configuration, please refer to the “Configuring Virtual Disks / LUNs” section below.

Configuring OMR on Dell|EMC Fibre Channel Storage with Dual HBAs and Dual Fibre Channel Switch Connections Figure 4 illustrates the fiber cabling of the two-node PowerEdge cluster hosting the OMR and the CX3 storage array where the data resides. Each CX3 storage array has two storage processors (SP), called SPA and SPB, which can access all of the disks in the system. Similar to the MD3000 storage, the physical disks in the CX3 storage array provide the physical storage capacity for the Oracle RAC database. Before data can be stored, the CX3 physical disks must be configured into components, known as RAID groups and LUNs. A RAID group is a set of physical disks that are logically grouped together. Each RAID group can be divided into one or more LUNs, which are logical entities that the server uses to store data. The RAID level of a RAID group is determined when binding the first LUN within the RAID group. It is


recommended to bind one LUN per RAID group for database workloads to avoid disk spindle contention.1 For details on LUN configuration, please refer to the “Configuring Virtual Disks / LUNs” section below. In the CX3 array, the LUNs are assigned to and accessed by the Oracle cluster nodes directly through one storage processor. In the event of a storage processor port failure, traffic will be routed to another port on the same SP if the host is connected to more than one SP port and the EMC PowerPath multi path software is used. In the event of a storage processor failure, LUNs on the failed processor will trespass to the remaining storage processor. Both events could result in an interrupted service unless multiple I/O paths are configured between the Oracle RAC database hosts and the CX3 array. Therefore, it is crucial to eliminate any single point of failures within the I/O path. At the interconnect level, it is recommended that each node of the Oracle RAC database have two HBAs with independent paths to both storage processors. With the EMC PowerPath software installed on the host, I/O can be balanced across HBAs as well. It is also recommended that two Fibre Channel switches are used because in the event of a switch failure in a single Fibre Channel switch fabric environment, all hosts will lose access to the storage until the switch is physically replaced and the configuration restored.

Figure 4 - Cabling a SAN-Attached Fibre Channel Cluster

Configuring Virtual Disks / LUNs As discussed in the Sections “Configuring OMR on PowerVault MD3000 Storage with Dual SAS 5/E Controller Connections” and “Configuring OMR on Dell|EMC Fibre Channel Storage with Dual HBAs and Dual Fibre Channel Switch Connections” above, a Virtual Disk is a logical unit of physical disks presented to the host in the MD3000 storage, while in the CX3 storage, the equivalent term to a Virtual Disk is a LUN. The storage for an Oracle 10g RAC database can be divided into the following three areas:

• The first area is for the Oracle Cluster Registry (OCR), the Clusterware Cluster Synchronization Services (CSS) Voting Disk, and the Server Parameter File (SPFILE) for the Oracle Automatic Storage Management (ASM) instances. The OCR stores the details of the cluster configuration,


1 “Designing and Optimizing Dell/EMC SAN Configurations Part 1”, Arrian Mehis and Scott Stanford, Dell Power Solutions, June 2004. http://www.dell.com/downloads/global/power/ps2q04-022.pdf

http://www.dell.com/downloads/global/power/ps2q04-022.pdf


including the names and current status of the database, associated instances, services, and node applications, such as the listener process. The CSS Voting Disk is used to determine the nodes that are currently available within the cluster. The SPFILE for ASM instances is a binary file which stores the ASM instance parameter settings.

• The second area is for database data that are stored in the Oracle database physical files including datafiles, online redo log files, control files, SPFILE for the database instances, and temp files for the temporary tablespaces.

• The third area is for the Oracle Flash Recovery Area which is a storage location for all recovery-related files. The disk based database backup files are stored in the Flash Recovery Area. The Flash Recovery Area is also the default location for all archived redo log files.

It is a best practice to separate the above three storage areas onto their own Virtual Disks / LUNs on separate Disk Groups / RAID Groups. The separation can enable better I/O performance by ensuring these files do not share the same physical disks. Table 3 shows a sample Virtual Disk / LUN configuration with three Virtual Disks / LUNs for each of the three storage areas described above. Virtual Disk / LUN Minimum Size RAID Number of

Partitions Used For

First Virtual Disk / LUN 1024 MB 10, or 1 Three of 300 MB each

Voting disk, Oracle Cluster Registry (OCR), and SPFILE for ASM instances

Second Virtual Disk / LUN Larger than the size of your database

10, or 5 for read-only

One Data

Third Virtual Disk / LUN Minimum twice the size of your second LUN/Virtual Disk

10, or 5 for read-only

One Flash Recovery Area

Table 3 - Virtual Disks / LUNs for the Cluster Storage Groups / RAID Groups.

Figure 5 illustrates a sample disk group and virtual disk configuration on a MD3000 array which separates the three storage areas on distinct physical disks. As the need for the storage increases, additional MD1000 expansions enclosures can be added to the storage subsystem. With the use of the Oracle Automatic Storage Management (ASM), expansion of the data storage area and the Flash Recovery Area can be simple and quick.

Figure 5 - Separation of Disk Groups and Virtual Disks within a PowerVault MD3000 Storage Array Figure 6 illustrates a sample RAID group and LUN configuration on a Dell|EMC CX3-20 storage with two Disk Array Enclosure (DAE). Similar to the MD3000 disk layout, there are separate partitions for the three storage areas described in Table 3. Spindles 0 through 4 in the Housing 0 of the CX3-20 contain the operating system for the storage. These spindles are also used during power outage to store the system cache data. It is not recommended to use the operating system spindles for other purposes.

Figure 6 - Separation of RAID Groups and LUNs within a Dell|EMC CX3-20 Storage Array



AID ting Disk, and SPFILE, data, and Flash Recovery Area reside should be

onfigured with RAID 10.

ance due to the reading and writing of parity blocks in addition to the ading and writing of database data.

figured at

“Configuring Shared Storage for the Database using the ASM Library Driver” section below.

onfiguring Fully Redundant Ethernet Interconnects

ar

the

el nect

iguration, with redundant private NICs on each server and redundant private network itches.

on

he Link ggregation Group, where one or more links are provided between the switches themselves.

n configuring NIC

aming, please refer to the “Configuring the Private NIC teaming” section below.

RAID 10 is considered the optimal choice for Oracle 10g RAC virtual disk / LUN implementation because it offers fault tolerance, greater read performance, and greater write performance.2 The disk group / Rgroup on which the OCR, Voc Because additional drives are required to implement RAID 10, it may not be the preferred choice for all applications. In these cases, RAID 1 can be used as an alternative for the disk group / RAID group for the OCR, Voting Disk, and SPFILE, which provides protection from drive hardware failure. However, RAID0 should never be considered as an option as this configuration does not provide any fault tolerance. Forthe disk group / RAID group of the virtual disk / LUN for the data storage area, RAID 5 provides a cost effective alternative especially for predominantly read-only workloads such as a data warehouse database. However, RAID 5 is not suitable for heavy write workloads, such as in an OLTP database, as RAID 5 can have significantly lower write performre Each virtual disk / LUN created in storage will be presented to all the Oracle RAC hosts and conthe OS level. For details on the shared storage configuration at the OS level, please refer to the

Server Configuration for the Oracle Management Repository

C Each Oracle 10g RAC database server needs at least two network interface cards (NICs), one for the external interface and one for the private interconnect network. The servers in an Oracle RAC are bound together using cluster management software called Oracle Clusterware, which enables the servers to appeas though they are a single server. Servers in the cluster communicate with each other using a dedicated private network also known as the cluster interconnect. One of the servers in the RAC cluster is assigned as the master node. In the event of a interconnect NIC failure in a single interconnect NIC environment, the server lost communication to the master node, and the master node will initiate recovery of the failed database instance on the server. In the event of a network switch failure in a single private network switch environment, a scenario will result equivalent to the failure of every single node in the cluster except fordesignated master node. The master node will then proceed to recover all of the failed instances in the cluster before providing a service from a single node which will result in a significant reduction in the levof service available. Therefore, it is recommended the implementation of a fully redundant interconnetwork conf

3sw Figure 7 illustrates the CAT 5E/6 Ethernet cabling of a fully redundant interconnect network configuratiof a two-node PowerEdge RAC cluster, with two private NICs on each server, and two private network switches. For this type of redundancy to operate successfully, it requires the implementation of tA To implement a fully redundant interconnect configuration requires the implementation of NIC teaming software at the operating system level. This software operates at the network driver level to provide twophysical network interfaces to operate underneath a single IP address.4 For details ote

2 “Pro Oracle Database 10g RAC on Linux”, Julian Dyke and Steve Shaw, Apress, 2006. 3 Dyke and Shaw, op. cit. 4 Dyke and Shaw, op. cit.

Figure 7 – Ethernet Cabling a Fully Redundant Private Interconnect Network

Configuring Dual SAS 5/E Controllers for MD3000 Storage As illustrated in Figure 3, it is recommended that two SAS 5/E controllers be installed on each of the PowerEdge server hosting an Oracle RAC database to provide redundant links to the PowerVault MD3000 storage array. Using dual SAS 5/E controllers protects against potential data loss or corruption if either a SAS 5/E controller or MD3000 storage controller fails.

Configuring Dual HBAs for Dell|EMC storage As illustrated in Figure 4, it is recommended that two HBAs be installed on each of the PowerEdge server hosting an Oracle RAC database because in the event of a HBA failure in a single HBA fabric environment, the host will lose access to the storage until the failed HBA is physically replaced. Using dual HBAs provides redundant links to the CX3 storage array.

Configuring Dual NICs for Private Network As illustrated in Figure 5, it is recommended that two private NICs be installed on each of the PowerEdge servers hosting an Oracle RAC database to provide redundant private network links. In the event of a NIC failure in a single private NIC environment, Oracle Clusterware will remove the node from the cluster.

Server Configuration for the Oracle Management Service (OMS)

Configuring Network requirement It is recommended that the OMS host and OMR host be located in close proximity to each other to avoid network bottlenecks. Ideally, the round trip network latency between the two should be less than one millisecond.



Software Configuration - Dell Solution for Oracle Enterprise Manager 10g Grid Control R3

Operating System Configuration

Configuring File System Partitions on Red Hat Enterprise Linux 4.4 When installing the operating system for the servers hosting the OMR and OMS, use the following partitioning scheme:

Partition Name Partition Size (in KB) /boot 256 / 10240 /tmp 4096 /usr 5120 /home 8192 /opt/oracle 12288 /var 2048 swap 8192

Table 4 - RHEL 4.4 File Partitions for OMS and OMS

Configuring the Private NIC Teaming As mentioned in the Section “Configuring Fully Redundant Ethernet Interconnects” above, two physical private NICs are installed on each of the Oracle RAC cluster server, to help guard against private network communication failures. Therefore, it is required to use NIC teaming software to bond the two private network interfaces together to operate under a single IP address. The NIC teaming software provides failover functionality. If a failure occurs, affecting one of the NIC interfaces – examples include switch port failure, cable disconnection, or failures of the NIC itself – network traffic is routed to the remaining operable NIC interface. Failover occurs transparently to the Oracle RAC database with no network communication interruption or changes to the private IP address.

Configuring the Same Public Network Interface Name It is important to ensure that all nodes within an Oracle 10g RAC cluster have the same network interface name for the public interface. For example, if “eth0” is configured as the public interface on the first node, then “eth0” should also be selected as the public interface on all of the other nodes. This is required for the correct operation of the Virtual IP (VIP) addresses configured during the Oracle Clusterware software installation.5

Configuring SSH and RSH During the installation of Oracle 10g RAC software, the Oracle Universal Installer (OUI) is initiated on one of the node of the RAC cluster. OUI needs to copy files to and run commands on the other hosts in the cluster. In order to allow OUI to do that, the secure shell (SSH) and remote shell (RSH) must be configured, so no prompts or warnings are received when connecting between hosts via SSH or RSH as the

5 Dyke and Shaw, op. cit.


oracle user. To prevent unauthorized users from accessing the systems, it is recommended that RSH be disabled after the Oracle software installation

Configuring Shared Storage for the Database using the ASM Library Driver Oracle Automatic Storage Management (ASM) is a feature of Oracle Database 10g that provides the database administrator (DBA) with a simple storage management interface that is consistent across all server and storage platforms. ASM virtualizes the database storage into disk groups. ASM distributes data evenly across all disks within a disk group to optimize performance and utilization. ASM enables the DBA to change the storage configuration without having to take the database offline. ASM automatically rebalances files across the disk group after disks have been added or dropped.6

As discussed in the Section “Configuring Virtual Disks / LUNs” above, two virtual disks / LUNs are created for the data storage area, and the Flash Recovery Area, respectively. It is recommended that these two virtual disks / LUNs be configured as ASM disks to benefit from the capabilities of ASM.

Installing Dell OpenManage Server Administration (OMSA) 5.2 on Red Hat Enterprise Linux 4.4 x86_64 OMSA is available as a set of downloadable tarballs containing the RPM files, installation scripts, and documentation in a tarzipped format. The package includes a script called “setup.sh” which must be executed for installation of OMSA 5.2. Before installing the OMSA 5.2 software, two steps must be completed. First, the following 32-bit RHEL 4.4 rpms should be installed on your system:

o pam-0.77-66.17.i386.rpm o ncurses-devel-5.4-13.i386.rpm o ncurses-5.4-13.i386.rpm o libxml2-2.6.16-6.i386.rpm o cracklib-dicts-2.7-29.i386.rpm o cracklib-2.7-29.i386.rpm and o audit-libs-1.0.14-1.EL4.i386.rpm.

Second, the /etc/snmp/snmpd.conf configuration file should be backed up because the OMSA 5.2 installation makes changes in this configuration file. To verify OMSA5.2 installation, check the version of OMSA installed on the monitored server with the command ‘omreport about’ and check the timestamp for the /etc/snmp/snmpd.conf file.

Oracle Enterprise Manager 10g Grid Control R3 Configuration

Configuring the RAC database that will support the OMR It is recommended that the OMR be placed in an Oracle 10g RAC database in order to make it highly available. This is accomplished by installing a dual-node Oracle RAC Cluster (with db_block_size parameter set to 8192) and creating a RAC database before installing the Oracle Enterprise Manager Grid Control software on the system. Dell provides detailed directions for deploying Oracle 10g RAC with Deployment Guides and Deployment CD images available from www.dell.com/oracle.

6 “Oracle Database 10g – Automatic Storage Management Overview”, Oracle TechNet. http://www.oracle.com/technology/products/manageability/database/pdf/asmov.pdf

http://www.dell.com/oracle

http://www.oracle.com/technology/products/manageability/database/pdf/asmov.pdf


Configuring DBMS_SHARED_POOL package for OMR After the RAC database which will be used as the OMR has been successfully created, it is recommended that the DBMS_SHARED_POOL package be installed; this package can help improve the throughput of the Management Repository. If the package has not been installed at the time of the database creation, the Oracle Universal Installer for Oracle Enterprise Manager will display an error message which will prompt the execution of this package before the installation. To determine if the DBMS_SHARED_POOL package has been installed, login to the database and execute the following query: Select count(*) from dba_objects where OWNER = 'SYS' AND object_name = 'DBMS_SHARED_POOL' AND object_type IN (‘PACKAGE’,'PACKAGE BODY') ; This query should return a count of 2. To install the DBMS_SHARED_POOL package, execute the following script: <DB_HOME>/rdbms/admin/dbmspool.sql

Configuring Database initialization parameters for the OMR As part of the OMR database creation, it is recommended that the database parameters for the Oracle 10g RAC database be set as per the values provided in the table below. For more information on setting up Oracle database parameters, please refer to the Oracle Enterprise Manager Grid Control Installation and Basic Configuration 10g Release 3 (10.2.0.3.0) guide.

DB Parameter Recommended Value 1. job_queue_processes 10 2. db_block_size 8192 3. timed_statistics TRUE 4. open_cursors 300 5. session_cached_cursors 200 6. aq_tm_processes 1 7. compatible (default) 8. undo_management AUTO 9. undo_retention 10800 10. undo_tablespace Any valid tablespace name 11. processes 150 12. log_buffer 1048576 13. statistics_level TYPICAL 14. _b_tree_bitmap_plans false (hidden parameter)

Table 5 - Recommended OMR Database Parameters

Configuring the Oracle Management Service (OMS) The Oracle Management Service is installed with and deployed on Oracle Application Server using the Oracle Universal Installer by executing the runInstaller from the installation media. The installation type is “Installing Enterprise Manager 10g Grid Control Using an Existing Database”. This option installs Enterprise Manager Grid Control onto the host machine and creates the Management Repository on the preconfigured Oracle RAC database. For further details on these steps refer to the Oracle Enterprise Manager Grid Control Installation and Basic Configuration 10g Release 3 (10.2.0.3.0) guide.


Specifying Repository Database Configuration during OMS installation During the operation of the Oracle Universal Installer when installing the repository database, the user will be queried to specify connection details. It is recommended to use the values provided in the table below:

Database Hostname VIP for the one Node of the RAC cluster

Port 1521

Service/SID service name of the RAC cluster or Instance SID

SYS password Password for SYS user

Management Tablespace Location Datafile path for management tablespace

Configuration Data Tablespace Location Datafile path for configuration data tablespace

Table 6 - Repository Database Connection Details

Once installation is complete, the $OMS_HOME/sysman/config/emoms.properties configuration file will contain the connect string for the node that was specified during the installation. The emoms.properties file may require editing in order to make the OMS aware of the second node. To help ensure that failover/high availability mode is enabled, the connect string in emoms.properties must be verfied for a two-node RAC configuration. If the file requires modification, stop the OMS, change the configuration and restart OMS.

Configuring Additional Oracle Management Agents (OMA) Oracle Enterprise Manager uses an Oracle Management Agent (OMA) on the OMS host to enable communications with the management agents installed on the target database hosts. The OMAs on the target hosts communicate with the OMA on the OMS server to provide data for monitoring the nodes. Before starting the installation of OMA on a target database host, verify that the target database hosts can ping the OMS server and that the OMS server can ping the target database host. While there are several ways to install OMA, a recommended option is to use the Oracle Universal Installer by executing “runInstaller” from the installation media and selecting the last option “Additional Management Agent”. During the installation, on the screen “Specify Oracle Management Service Location,” be sure to provide the correct hostname for the OMS Server and Port.

Configuring Integration of Oracle Enterprise Manager 10g Grid Control R3 with OMSA 5.2 Once all software installations have been successfully completed, SNMP must be configured to allow OMSA to interface with the Oracle Enterprise Manager agent (OMA) on each system. To enable the SNMP connection, create or modify the entry ‘rocommunity public 127.0.0.1’ for the localhost IP address in the /etc/snmp/snmpd.conf file; this enables OMSA to communicate with OMA running on the same server. Then restart the snmpd service with the command “service snmpd restart” or reboot the server. The command “chkconfig –level 35 snmpd on” causes the snmpd process to startup on system boot. Once these commands are executed, the PowerEdge server hardware metrics covering components such as Fans, Memory, and Power Supplies will be available in Oracle Enterprise Manager.


Configuration Deliverables List - Dell Solution for Oracle Enterprise Manager 10g Grid Control R3 on Red Hat Enterprise Linux 4 Oracle provides a number of recommendations for selecting the size of your systems management configuration. Here are some general guidelines for the management server and management repository:

MANAGEMENT SERVER MANAGEMENT REPOSITORY

Deployment Size

Hosts Memory/ Host (Gb) Hosts Memory/

Host (Gb)

Total Repository

Storage (Gb)

Small (100 monitored

targets) 1 2 Shares host with Management

Server 10

Medium (1,000 monitored

targets) 1 4 2 4 30

Large (10,000 monitored

targets) 2 8 2 6 100

Table 7 - Management Server and Repository Sizing


Following is the Solution Deliverables List (SDL) for the Dell Solution for Oracle Enterprise Manager 10g Grid Control R3 on RHEL 4.4. It contains a detailed listing of the server and storage hardware

configurations, firmware, driver, OS and database versions.

Minimum Hardware/Software Requirements (For details, see below)

Validated Component(s)

Minimum Single Node (OMS/OMR)

Configuration

Minimum Multiple Node

(OMR) Configuration

PowerEdge Nodes 1950, 2950 1 2

Memory All valid PowerEdge

1950 and 2950 memory configurations)

8GB (per node) 8GB (per node)

Dell | EMC FC Storage CX3-20 1 2

PowerVault Storage PowerVault MD3000 1 1

Fibre Channel Switch

Brocade SW200E, SW4100 2

2 (8 port switches for 2-4 nodes, and

16 port switches for 5 or above nodes) 1

HBAs QLE2460 2 single-port HBAs (Per Node)2

2 single-port HBAs (Per Node)2

Ethernet Ports Intel or Broadcom Gigabit NICs 3 (Per Node) 3 3 (Per Node) 3

Ethernet Switches (For Private Interconnect)

Gigabit-only Switches N/A 2

Raid Controllers (For internal storage only)

PERC 5/e, PERC 5/i 1 (Per Node) 1 (Per Node)

Internal Drive All valid PowerEdge

1950 and 2950 internal storage configurations

73 Gig/node 73 Gig/node

Oracle Software & Licenses

OMR : Oracle 10g R2 10.2.0.1 Enterprise

Edition(Base) + Oracle Patchset 10.2.0.3

OMS : Oracle Enterprise Manager 10g R3

Enterprise Edition RAC

Operating System Red Hat Enterprise Linux 4 Update 4 + errata

Recommended Support Contract

Dell Gold or Platinum Plus Enterprise Support

Table 8 - Solution Minimal Hardware/Software Requirements

Notes: 1: Assumes no additional ports required for other functions on the switch. 2: Two single port HBAs are recommended. 3: Assumes one NIC for public network, and two bonded NICs for private network.


Reference Configuration Details

Validated Servers

Model

BIOS ESM/BMC Firmware

Internal SCSI RAID Firmware / Driver

Notes

PE1950 1.4.0 or higher

v1.33 or higher

PERC 5i –5.1.1-0040/00.00.02.03-RH1

Intel Dual Core

Processors only

Dell PowerEdge Servers PE2950 1.4.0

or higher v1.33 or higher

PERC 5i –5.1.1-0040/00.00.02.03-RH1

Intel Dual Core

Processors only

PERC 5/e, 5/i Firmware = v5.1.1-0040; Driver = version 00.00.02.03-

RH1 Network Interconnect

Ethernet Switches All Dell Gigabit Ethernet Switches Intel NIC Drivers (100) Driver = (e1000) 7.0.33-k2 NAPI Intel NIC Drivers (1000MT) Driver = (e1000) 7.0.33-k2 NAPI Broadcom NIC Drivers (5708) Driver = (bnx2)1.4.38 NIC Bonding NIC Kernel bonding for QU4 Version = 2.6.3

Fibre Channel Host Bus Adapter (HBA) Qlogic HBA QLE2460 BIOS = 1.08 ; Firmware = 4.00.18; Driver = (qla2400)

version 8.01.04-d8 Fibre Channel Switches

Brocade Fibre Channel Switch (SW4100, SW200E)

Firmware = v5.2.1 or higher

Direct Attached SAS Storage Dell PowerVault Storage MD3000; Firmware = v.06.17.77.60, A01

MD1000; Firmware = v.A.03, A03 Fibre Channel Storage

Validated Storage Arrays ( with Software)

Dell | EMC Storage Subsystem: CX3-20f(Release 24 or later)

Database Software Oracle Database (OMR) 10g R2 10.2.0.1 Enterprise Edition (Base) + 10.2.0.3

(PatchSet) Oracle Enterprise Manager 10g R3 ASMLib Oracleasm-2.6.9-42.0.8 Operating system RHEL4 QU4 (kernel-2.6.9-42.0.8 ELsmp, 2.6.9-

42.0.8.ELlargesmp) DKMS Version 2.0.13-1 EMC PowerPath 5.0.0-156 (available at www.emc.com)

Table 9 - Solution Detailed Firmware, Driver and Software Versions


Conclusion Dell Solutions for Oracle Enterprise Manager 10g Grid Control R3 are designed to simplify operations, improve utilization and cost-effectively scale as your needs grow over time. This reference architecture white paper provides a blueprint for setting up Oracle Enterprise Manager 10g Grid Control R3 on Dell PowerEdge servers and Dell PowerVault and Dell|EMC storage arrays. The best practices described here are intended to help achieve optimal performance of Oracle Enterprise Manager 10g R3. To learn more about deploying Oracle 10g Solutions on PowerEdge server and Dell storage, please visit www.dell.com/oracle or contact your Dell representative for up to date information on Dell servers, storage and services for Oracle solutions.


Tables and Figures Index Table 1 - Compatibility Matrix for OEM Grid Control Framework............................................................... 7 Table 2 - Reference Configuration for OEM Grid Control Framework ......................................................... 7 Table 3 - Virtual Disks / LUNs for the Cluster Storage Groups / RAID Groups. ........................................ 11 Table 4 - RHEL 4.4 File Partitions for OMS and OMS ............................................................................... 15 Table 5 - Recommended OMR Database Parameters .................................................................................. 17 Table 6 - Repository Database Connection Details ...................................................................................... 18 Table 7 - Management Server and Repository Sizing .................................................................................. 19 Table 8 - Solution Minimal Hardware/Software Requirements ................................................................... 20 Table 9 - Solution Detailed Firmware, Driver and Software Versions......................................................... 21 Figure 1 - Architectural Overview of Enterprise Manager Grid Control Direct-Attached SAS Storage

Configuration ................................................................................................................................. 5 Figure 2 - Architectural Overview of Enterprise Manager Grid Control Fibre Channel Storage

Configuration ................................................................................................................................. 6 Figure 3 - Cabling a Direct Attached MD3000 Storage Array with two MD1000 expansion units............... 9 Figure 4 - Cabling a SAN-Attached Fibre Channel Cluster ......................................................................... 10 Figure 5 - Separation of Disk Groups and Virtual Disks within a PowerVault MD3000 Storage Array ..... 12 Figure 6 - Separation of RAID Groups and LUNs within a Dell|EMC CX3-20 Storage Array................... 12 Figure 7 – Ethernet Cabling a Fully Redundant Private Interconnect Network ........................................... 14

References 1. “Designing and Optimizing Dell/EMC SAN Configurations Part 1”, Arrian Mehis and Scott Stanford,

Dell Power Solutions, June 2004. http://www.dell.com/downloads/global/power/ps2q04-022.pdf

2. “Pro Oracle Database 10g RAC on Linux”, Julian Dyke and Steve Shaw, Apress, 2006. 3. “Oracle Database 10g – Automatic Storage Management Overview”, Oracle TechNet.

http://www.oracle.com/technology/products/manageability/database/pdf/asmov.pdf 4. “Benchmark Factory for Databases”, Quest Software.

http://www.quest.com/Quest_Site_Assets/PDF/Benchmark_Factory_5_TPCH.pdf

http://www.dell.com/downloads/global/power/ps2q04-022.pdf

http://www.oracle.com/technology/products/manageability/database/pdf/asmov.pdf

http://www.quest.com/Quest_Site_Assets/PDF/Benchmark_Factory_5_TPCH.pdf

white paper dell reference configuration deploying oracle...

Documents