maximizing performance on a - · pdf filereference configuration for oracle 11g se rac on dell...

Reference Configuration for Oracle 11g SE RAC on Dell EqualLogic PS4000XV Storage Arrays 1

Maximizing Performance on a Cost Effective Oracle® 11g SE RAC Database on Dell™ EqualLogic™ PS4000XV iSCSI Storage A Dell Technical White Paper

Database Solutions Engineering By Wendy Chen, Naveen Iyengar, and David Mar Dell Product Group July 2009


THIS WHITE PAPER IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS AND TECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND. © Copyright 2009 Dell, Inc. All rights reserved. Reproduction in any manner whatsoever without the express written permission of Dell Inc. is strictly forbidden. For more information, contact Dell. Dell, the Dell logo, PowerEdge, and EqualLogic are trademarks of Dell Inc; Intel and Xeon are registered trademarks of Intel Corporation in the U.S. and other countries; Microsoft, SQL Server, Windows, and Windows Server are either trademarks or registered trademarks of Microsoft Corporation in the United States and/or other countries; Oracle is a registered trademark of Oracle Corporation and/or its affiliates; Red Hat and Red Hat Enterprise Linux are registered trademarks of Red Hat, Inc. Other trademarks and trade names may be used in this document to refer to either the entities claiming the marks and names or their products. Dell disclaims proprietary interest in the marks and names of others.


ABSTRACT ..................................................................................................................................... 5

INTRODUCTION ............................................................................................................................. 5

DELL SOLUTIONS FOR ORACLE 11G DATABASE ............................................................................... 6

ORACLE 11G DATABASE STANDARD EDITION OVERVIEW ................................................... 6

ORACLE 11G LICENSING OPTIONS .................................................................................................. 6 FEATURE AVAILABILITY BY EDITION ................................................................................................. 7

DELL EQUALLOGIC PS4000 STORAGE SYSTEMS OVERVIEW ............................................... 7

EASE OF USE ................................................................................................................................ 7 HIGH PERFORMANCE ..................................................................................................................... 7 ENTERPRISE-CLASS FEATURES ...................................................................................................... 8 LOW COST OF OWNERSHIP ............................................................................................................ 8

ORACLE 11G DATABASE SE RAC ARCHITECTURE OVERVIEW ............................................ 8

SEAMLESS SCALABILITY ................................................................................................................. 9

HARDWARE CONFIGURATION .................................................................................................. 10

STORAGE CONFIGURATION .......................................................................................................... 10 Configuring Dell EqualLogic PS4000XV iSCSI Storage Connections .................................. 10 Configuring Volumes ............................................................................................................. 12 Configuring Challenge Handshake Authentication Protocol (CHAP) ................................... 13

ISCSI SAN GIGABIT ETHERNET SWITCH CONFIGURATION ............................................................. 14 Configuring iSCSI SAN Network ........................................................................................... 14

SERVER CONFIGURATION............................................................................................................. 15 Configuring Fully Redundant Ethernet Interconnects ........................................................... 15 Configuring Multiple Ethernet Interfaces for iSCSI Storage Area Networks ......................... 15

SOFTWARE CONFIGURATION ................................................................................................... 16

OPERATING SYSTEM CONFIGURATION .......................................................................................... 16 Configuring Host Access to iSCSI Volumes ......................................................................... 16 Configuring the Oracle Private Network NIC Teaming ......................................................... 17 Configuring Network Requirements on All Nodes ................................................................ 17 Configuring Host Equivalence .............................................................................................. 17 Configuring Shared Storage for Oracle Clusterware Using Block Device ............................ 17 Configuring Shared Storage for the Database Using the ASM Library Driver ...................... 18

ORACLE 11G R1 CONFIGURATION ................................................................................................ 18 Installing Oracle Clusterware and Database Software ......................................................... 18 Installing Recommended Oracle Patches ............................................................................. 18

PERFORMANCE CAPABILITIES OF ORACLE 11G SE RAC ................................................... 19

TEST TOOLS AND CONFIGURATION ............................................................................................... 19 Quest Benchmark Factory TPC-C ........................................................................................ 19 Hardware and Software Configuration .................................................................................. 19 Performance Monitoring ........................................................................................................ 20

TUNING ORACLE MEMORY SIZE .................................................................................................... 23 TESTING RAC SCALABILITY WITH 24 GB RAM PER NODE ............................................................. 24

CPU Performance Analysis .................................................................................................. 24 Memory Utilization Analysis .................................................................................................. 26 Storage System Performance Analysis ................................................................................ 26 Examining both CPU and Memory Utilization ....................................................................... 27 Overall Performance Analysis .............................................................................................. 28

TESTING RAC SALABILITY WITH 48 GB RAM PER NODE ............................................................... 28

CONTENTS


CPU Performance Analysis .................................................................................................. 29 Memory Utilization Analysis .................................................................................................. 31 Examining both CPU and Memory Utilization ....................................................................... 32 Storage System Performance Analysis ................................................................................ 32 Overall Performance Analysis .............................................................................................. 33

PERFORMANCE CHARACTERISTICS SUMMARY ............................................................................... 34

SOLUTION DELIVERABLES LIST FOR ORACLE 11G ON DELL EQUALLOGIC PS4000XV ISCSI STORAGE........................................................................................................................... 36

CONCLUSION ............................................................................................................................... 38

TABLES AND FIGURES .............................................................................................................. 39

REFERENCES .............................................................................................................................. 39


Abstract Customers rely on Oracle databases for their mission critical workloads, but sometimes consider alternatives because of the cost of the enterprise edition database. With the significantly lower cost of the Oracle Standard Edition (SE) database, with Oracle Real Application Cluster (RAC) capabilities extending to 4-socket processors, Oracle SE RAC offers database customers a lower cost option that has enterprise level performance. Customers that are looking to implement an Oracle RAC database can utilize the Oracle SE, Red Hat® Enterprise Linux® and commodity hardware in order to lower their total cost of ownership (TCO) of their enterprise data. This paper presents a “pay-as-you-grow” methodology to build an Oracle SE RAC configuration that can scale up to 4 single-socket servers. It showcases the performance characteristics and scalability of incrementally adding RAC nodes.

Introduction Oracle 11g Database Standard Edition is built from the same code base as the Oracle 11g Enterprise Edition, and is ideally suited to the needs of small to medium-sized business (SMB). Oracle 11g database enables business applications to take advantage of performance, reliability, security and scalability at a lower cost. Oracle 11g database includes the Oracle Real Application Clusters (RAC) for enterprise-class availability. Dell’s newest PowerEdge™ 11th generation servers feature energy-tuned technologies designed to reduce power consumption while increasing performance and capacity. The Lifecycle Controller (LC) components simplify administrator tasks by performing a complete set of provisioning functions such as system deployment, system updates, hardware configuration and diagnostics from a single intuitive interface called the Unified Server Configurator (USC) in a pre-OS environment. The new Dell Management Console (DMC) delivers a single view and a common data source into the entire infrastructure. DMC helps to reduce or eliminate manual processes so that less time and money is spent on maintenance and more can be spent on strategic uses of technology. The PowerEdge R610 1U rack server features Intel® 5500 series processors. This processor features quad-core processing to maximize performance, and performance/watt, for data center infrastructures and highly dense deployments. Two other notable features that benefit multi-threaded demanding database workloads are:

• Support for CPU turbo mode (on certain SKUs) that increases CPU frequency if operating below thermal, power, and current limits

• Simultaneous multi-threading (hyper-threading) capability that increases application performance by delivering greater throughput and responsiveness

The PowerEdge R610 utilizes DDR3 memory that provides a high performance, high-speed memory interface capable of low-latency response and high throughput. Dell EqualLogic PS4000 is the newest addition to the PS Series family of iSCSI storage arrays. Designed for deployment in SMB environments, the PS4000 offers an entry-point to the PS Series providing easy administration, enterprise software, and virtualized architecture at an affordable price point. The PS4000 arrays support both SAS and SATA disk drives and include three product lines: the PS4000XV, PS4000X and PS4000E that are differentiated by the disk drive type and speed supported. The PS4000XV offers the best performance by supporting 15 K RPM SAS disks. PS4000XV is an ideal choice to deploy highly reliable and sustainable Oracle 11g RAC databases. This reference configuration white paper is intended to help IT professionals design and configure Oracle 11g RAC database solutions using Dell EqualLogic arrays and servers that apply best practices derived from laboratory and real-world experiences. This white paper documents the Dell recommended approach for implementing a tested and validated solution for Oracle 11g


RAC database on Dell EqualLogic PS4000XV iSCSI storage arrays, and the Dell PowerEdge 11th generation R610 servers running Enterprise Linux 5 Update 3.

Dell Solutions for Oracle 11g Database Dell solutions for Oracle 11g databases are designed to simplify operations, improve usability, and cost-effectively scale as your needs grow over time. In addition to providing server and storage hardware, Dell solutions for Oracle 11g include:

• Dell Configurations for Oracle – in-depth testing of Oracle 11g configurations for high-demand solutions; documentation and tools that help simplify deployment

• Integrated Solution Management – standards-based management of Dell solutions for Oracle 11g that can lower operational costs through integrated hardware and software deployment, monitoring, and updates

• Oracle Licensing – multiple licensing options that can simplify a customer purchase • Dell Enterprise Support and Infrastructure Services for Oracle – offerings for the

planning, deployment, and maintenance of Dell solutions for Oracle 11g For more information concerning Dell solutions for Oracle 11g database, please visit dell.com/oracle.

Oracle 11g Database Standard Edition Overview Oracle 11g Licensing Options Oracle 11g database is available in multiple editions, each tailored to meet the development and deployment needs of different sizes of IT organizations. Here are the two most relevant to this white paper: • Oracle Standard Edition (SE) is available for single or clustered servers supporting up a

maximum of four CPU sockets. It includes selected, but not all, features that come with the Oracle Enterprise Edition (EE).

• Oracle Enterprise Edition (EE) is available for single or clustered servers with no limit on the

maximum number of CPU sockets. It contains all of the Oracle database components, and can be further enhanced with the purchase of options and packs.

Oracle 11g SE includes Oracle Real Application Clusters (RAC) for enterprise-class availability. When used with Oracle RAC in a clustered server environment, Oracle 11g Standard Edition requires the use of Oracle Clusterware. Automatic Storage Management (ASM) must be used to manage all database-related files. No other cluster software can be installed on the system, including OCFS (Oracle Cluster File System), OCFS2, third-party clusterware, third-party cluster volume managers, or third-party cluster file systems. Oracle 11g SE can support clustered servers with up to four CPU sockets in the cluster. Oracle Database 11g Standard Edition is available from Dell on versions of Windows, Linux and Solaris. Using the same proven concurrency techniques as Oracle 11g EE, Oracle 11g SE ensures maximum throughput for all workloads. With the implementation of Oracle RAC, workloads are automatically balanced across the available servers in the cluster to ensure maximum hardware utilization. RAC also protects the database from machine failures. When a node in the cluster fails, the database continues to run in a degraded mode on the surviving nodes of the cluster.


Feature Availability by Edition Each edition of the Oracle 11g database includes features to meet the varying business environment requirements. SE contains a different subset of database features that are all included in EE. Additionally, a number of advanced options and packs can be purchased with EE at an extra cost, to meet the most demanding requirements of mission-critical databases. A number of database features that come with the Oracle 11g EE are not available in SE. Some of these features are highlighted below:

• Data guard • Rolling upgrades • Online index and table organization • Parallel backup and recovery • Table spaces point-in-time recovery • Flashback table / transaction / database • Parallel query / statistics gathering / index builds / data pump export and import • Transportable table spaces • Oracle connection manager • Infiniband support

A complete list of feature availability by edition can be found in the Oracle Database Licensing Information 11g Release 1 (11.1) documentation. Before deploying your databases on Oracle SE, make sure that you do not require one of non-supported features. If you do require one of these features, it is recommended that you deploy Oracle EE.

Dell EqualLogic PS4000 Storage Systems Overview The Dell EqualLogic PS4000 Series of iSCSI SAN arrays are designed to bring enterprise-class features, intelligence, automation, and reliability to SMB storage deployments. The PS4000 Series addresses the storage needs of SMB with simplified administration, rapid deployment, and an affordable price while providing a full set of enterprise-class data management and data protection software features, self-optimized high performance, and pay-as-you-grow scaling.

Ease of Use By including SAN configuration features and capabilities that sense network connections, automatically build RAID sets, and conduct system health checks the PS4000 Series is designed to be installed and configured in less than one hour. The Dell EqualLogic SAN HeadQuarters software tool enables consolidated performance and event monitoring across multiple EqualLogic SAN groups. SAN HeadQuarters helps administrators to better understand storage performance and capacity usage, and quickly be informed of events and potential issues.

High Performance All Dell EqualLogic PS Series iSCSI storage arrays are built on a patented peer storage architecture, where all arrays in a storage pool are designed to work together to provide disk capacity and evenly distribute the load. This architecture enables the arrays to continuous monitor resource and performance, and automatically moves data pages to maximize performance.


Enterprise-class Features The PS4000 Series storage arrays include advanced software features with no additional software licensing fee. These features enable easy management, data protection, and storage virtualization. Examples of the software features include PS Group Manager, SAN HeadQuarters multi-group monitoring, multi-path I/O, Auto-Snapshot Manager, and automatic load balancing.

Low Cost of Ownership The Dell EqualLogic PS4000 Series storage system offers an entry-point to the PS Series at an affordable price point. The PS4000 Series drives down the total cost of SAN storage with an all-inclusive package of highly-reliable hardware and comprehensive software features. The PS4000 Series can expand to up to two PS4000 arrays in one storage group. To support higher performance and capacity scalability requirements, PS4000 arrays can be upgraded to PS6000 Series that is the high-end PS Series storage product.

Oracle 11g Database SE RAC Architecture Overview The Dell reference configuration for Oracle 11g SE on Dell EqualLogic PS4000XV iSCSI storage arrays is intended to validate the following solution components:

• Up to a four-node cluster comprised of Dell PowerEdge R610 servers • Dell EqualLogic PS4000XV iSCSI storage systems • Red Hat Enterprise Linux 5 Update 3 • Oracle 11g R1 (11.1.0.7) SE x86_64

An architectural overview of the Dell solution for Oracle 11g SE on Dell EqualLogic PS4000XV iSCSI storage arrays is shown in Figure 1. The sample architecture represents a two-node Oracle RAC comprised of the following components:

• Dell Optiplex™ desktop systems that will access data stored within the Oracle database • Client-server network made up of network controllers, cables, and switches • Dell PowerEdge R610 servers running Red Hat Enterprise Linux 5 Update 3 and Oracle

11g R1 RAC (11.1.0.7) • Redundant Dell gigabit Ethernet switches for the Oracle cluster interconnect network • Server-storage interconnect using redundant Dell PowerConnect™ 6248 gigabit Ethernet

switches • Two Dell EqualLogic PS4000XV iSCSI storage arrays configured as RAID 10.


Figure 1: Oracle 11g SE on Dell EqualLogic PS4000XV iSCSI Storage Architecture Overview

Seamless Scalability Oracle RAC offers a number of benefits when compared to a traditional single instance database:

• High availability – RAC is typically built to eliminate any single point of failure by implementing redundant components. In the event of a node or database instance failure, application connections can be automatically failed over to the surviving instances.

• Scalability - Multiple nodes in a RAC allows the database to scale beyond the limit of a single node database. Additional servers can be added into an existing RAC cluster when the workload increases and demands more processing power. Adding an additional node to or removing existing nodes from, a RAC can be done without any database downtime.

• Low cost of ownership - RAC systems are typically implemented on industry standard hardware that is less expensive than proprietary systems; this helps customers reduce the cost of purchase, maintenance, and support.

With the ability to support up to four sockets worth of compute resources in a cluster, Oracle 11g SE allows a customer to start with one single socket machine at an attractive price point, and simply scale up to four single socket machines as their business demands grow. This “pay-as-you-grow” methodology helps maximize hardware utilization and lowers the total cost of ownership. Scalability is generally measured by the throughput while increasing the workload; RAC scalability is dependent on the scalability of the hardware, storage, operating systems, database, and application. The scalability that can be achieved by Oracle RAC varies according to the design and configurations of the hardware and software.


Figure 2: "Pay-as-you-grow" Methodology to Scale up to Four Single Socket Servers

Hardware Configuration Storage Configuration

Configuring Dell EqualLogic PS4000XV iSCSI Storage Connections The Dell EqualLogic PS4000XV iSCSI storage array offers a highly-available hardware design that includes redundancy features to ensure that there is no single point of failure. Its components are fully redundant and hot swappable, including the 16 SAS disks, dual active/standby control modules, dual fan trays, and dual power supplies. As illustrated in Figure 3, the control modules are identified as control module 0 and control module 1. Each control module has two 1 gigabit Ethernet interfaces for the SAN network, labeled as ethernet0 and ethernet1. Each control module also has one 10/100 Ethernet management port to be used for management purposes only. Each control module is equipped with a 2 GB battery-backed, write-back cache to ensure cache coherency between the two control modules. In the event of a control module failure, the other control module takes over automatically with no disruption to users.

Figure 3: Recommended Dual Control Module Network Configuration Host servers can be attached to the PS4000XV through an IP storage area network (SAN) industry-standard gigabit Ethernet switch. Figure 3 shows the recommended network configuration for a dual control module PS4000XV array. This configuration includes two redundant Dell PowerConnect 6248 gigabit Ethernet switches to provide the highest network availability, and maximum network bandwidth. It is recommended that two gigabit Ethernet


switches are used, because if a switch fails in a single Ethernet switch environment all hosts will lose access to the SAN until the switch is physically replaced and the configuration restored. From each of the control modules, it is recommended that one gigabit interface connects to one Ethernet switch, and the other gigabit interface connect to the other Ethernet switch for high availability.

Figure 4: Cabling Two-Node Oracle Database 11g Standard Edition Real Application Clusters Database Figure 4 illustrates the cabling of a two-node PowerEdge cluster hosting the Oracle database and two PS4000XV storage arrays where the data resides. The blue colored cables are the iSCSI storage area network, and the red colored cables are the Oracle RAC private interconnect network. The black colored cables are the public network. The PS4000XV storage arrays provide the physical storage capacity for the Oracle 11g RAC database; host servers access the data through a single group IP address. A PS Series group is a storage area network comprised of one or more PS Series arrays. There can be up to two PS4000 Series arrays in a PS Series group. Each array in a group is referred to as a member. All members in a group are managed and accessed as a single storage system using the group IP address. A group or a member has a name assigned and each member must have a RAID level specified when it is initialized: RAID 5, RAID 6, RAID 10, or RAID 50. As illustrated in Figure 4, the group named oracle-group includes two PS4000XV members: oracle-member01 and oracle-member02. A PS Series storage group can be segregated into multiple tiers or pools; tiered storage provides administrators with greater control over how disk resources are allocated. At any one time, a member can be assigned to only one pool. It is easy to assign a member to a pool, and also to move a member between pools, with no impact to data availability. Pools can be organized according to different criteria, such as disk types or speeds,


RAID levels, or application types. In Figure 4, there is a pool with the name RAID-10 that consists of RAID 10 members. Before data can be stored, the PS4000XV physical disks must be configured into usable components, known as volumes. A volume represents a portion of the storage pool, with a specific size, access controls, and other attributes. A volume can be spread across multiple disks and group members, and is seen on the network as an iSCSI target. Volumes are assigned to a pool, and can be easily moved between pools, with no impact on data availability. In addition, automatic data placement and automatic load balancing occurs within a pool based on the overall workload of the storage hardware resources within the pool. For details on volume configuration for an Oracle RAC database, refer to the Configuring Volumes section. At the host level, in order to provide sufficient bandwidth to support two PS4000XV arrays it is recommended that each Oracle 11g RAC node have multiple gigabit NIC ports with independent paths to both gigabit Ethernet switches of the iSCSI SAN. By utilizing the Linux Device Mapper (DM) Multipathing on the cluster node, I/O can be balanced across NIC ports as well. As shown in Figure 4, each host server has separate connections to the redundant gigabit Ethernet switches and to the dual control modules. In this configuration, there is pathway redundancy at the host, at the two Ethernet switches, and at the dual RAID control modules. On each server the four NIC ports used for iSCSI traffic, as well as the two NIC ports used for Oracle private interconnect network, are configured on separate NIC cards to protect against a single PCI bus failure. At the Ethernet switch interconnection level, the bandwidth of the inter-switch link is critical to support high-performing applications. In the Oracle configuration shown in Figure 4, the two Dell PowerConnect 6248 gigabit Ethernet switches used for routing iSCSI traffic are stacked together with 48 Gbps bandwidth.

Configuring Volumes Oracle ASM is a feature of Oracle 11g that provides vertical integration of the file system and volume manager. ASM distributes the I/O load across all available resources to optimize performance, while removing the need for manual I/O tuning by spreading out the database files to avoid “hotspots.” ASM helps DBAs manage a dynamic database environment by allowing them to grow the database size without having downtime to adjust the storage allocation.

Oracle 11g RAC database storage can be divided into three shared storage areas: • Oracle Cluster Registry (OCR) and Cluster Synchronization Services (CSS) Voting Disk

shared storage. The OCR stores the cluster configuration details, including the names and current status of the database, associated instances, services, and node applications such as the listener process. The CSS Voting Disk is used to determine which nodes are currently available within the cluster. Unlike traditional database files, these files cannot be placed on the disks managed by ASM because they must be accessible before the ASM instance starts. These files can be placed on block devices that are shared by all the RAC nodes.

• Actual Oracle database that is stored in the physical files including the data files, online redo log files, control files, SPFILE for the database instances, and temp files for the temporary table spaces. The volumes(s) in this area are used to create the ASM disk group and managed by ASM instances.

• Oracle Flash Recovery Area shared storage that is an optional storage location for all

recovery-related files, as recommended by Oracle. If configured, the disk-based database backup files are all stored in the Flash Recovery Area. The Flash Recovery Area is also the default location for all archived redo log files, multiplexed copies of control files, and online redo log files. The size of the Flash Recovery Area will depend on what and how much is being stored.


Table 1 shows a sample volume configuration, with volumes for each of the three previously described shared storage areas. Volume Minimum Size RAID Number of

Partitions Used For OS Mapping

First Area Volume

1024 MB 10 Three of 300 MB each

Voting disk, Oracle Cluster Registry (OCR), and SPFILE for ASM instance

Three block devices, each for Voting Disk, OCR, and SPFILE

Second Area Volume(s)

Larger than the size of your database

10 One Data ASM disk group DATABASEDG

Third Area Volume(s)

Depends on the size and activity levels on the database

10 One Flash Recovery Area

ASM disk group FLASHBACKDG

Table 1: Volumes for the Oracle RAC Configuration RAID 10 is considered the optimal choice for Oracle 11g RAC LUN implementation because it offers fault tolerance, excellent read performance, and outstanding write performance. The PS4000XV array member where the data is allocated should be configured with RAID 10. In a PS Series storage group, access control records are used to control which hosts can access a volume. Each PS Series volume has a single list of access control records. In each record, you can specify an IP address, iSCSI initiator name, or Challenge Handshake Authentication Protocol (CHAP) user name (or any combination of the three). A server must match all the requirements in one record in order to access the volume. The most secure way to control access to your volumes is to use a combination of an IP address and a CHAP user name. For example, if a record includes both an IP address and a CHAP user name, a server must present the IP address and supply the CHAP user name and its associated password in order to match the record. Each storage volume will be presented to all the Oracle 11g RAC hosts and configured at the OS level. For details on the shared storage configuration at the OS level, refer to the “Configuring Host Access to the iSCSI Volumes,” “Configuring Shared Storage for the Oracle Clusterware,” and “Configuring Shared Storage for the Database Using the Automatic Storage Management (ASM)” sections. As previously discussed, the Flash Recovery Area is an optional disk-based recovery area that can be used to store files for backup and recovery operations. If users choose not to configure the disk-based Flash Recovery Area, PS Series arrays provide alternative methods to perform Oracle database disk-based backups: snapshots, clones, and replications. Snapshots enable point in time copies of volume data that can be used for backups. Snapshots are the most space-efficient method of volume protection, and utilize reserve space to hold the delta between the original base volume data and the snapshot data. Clones are a complete point-in-time copy of the base volume. The replication feature allows volume data to be replicated to a secondary or remote PS Series storage group.

Configuring Challenge Handshake Authentication Protocol (CHAP) As mentioned in the preceding section, the Challenge Handshake Authentication Protocol (CHAP) can be used to restrict host access to PS Series storage volumes. CHAP is an iSCSI authentication method that authenticates access between the storage volumes (targets) and the iSCSI initiators on the host servers. CHAP is an optional feature, and is not required to use iSCSI. However, with CHAP authentication, volume access can be restricted to hosts that supply


the iSCSI initiator with the correct user name and password. This information must match an access control record for the volume, in addition to an entry in a CHAP database, in order to gain access to the volume.

iSCSI SAN Gigabit Ethernet Switch Configuration

Configuring iSCSI SAN Network For the best performance, the iSCSI SAN network should be isolated from other networks by dedicating gigabit Ethernet switches for iSCSI traffic, or using VLANs to separate networks within a switch. As illustrated in Figure 1, the iSCSI network is physically isolated from the client network by using two dedicated Dell PowerConnect 6248 gigabit Ethernet switches. To achieve optimal performance, follow these SAN network guidelines for the PS Series iSCSI storage network:

• Use the Rapid Spanning Tree Protocol (RSTP) and enable the PortFast setting on the switch ports between switches. The Spanning Tree Protocol (STP) is a link management protocol that prevents loops in an Ethernet network by ensuring that only one active path exists between switches. Upon linkup, a switch performs a 30 to 50 second STP calculation to transition ports into forwarding or blocking state. STP can increase the time it takes to recover from a PS Series array control module failover or a network switch failure, so it is recommended to enable switch port settings that allow the immediate transition of the port into STP forwarding state upon linkup; this can reduce network interruptions that occur when devices restart. For example, the Dell PowerConnect 6248 gigabit Ethernet switch includes a feature called PortFast that immediately transitions a port into STP forwarding state upon linkup. It is also preferable to use Rapid Spanning Tree Protocol (RSTP) instead of STP. RSTP allows a switch port to bypass the STP listening and learning states, and quickly enter the STP forwarding state.

• Enable flow control on switch ports and NIC ports. When the data transmissions from

network senders exceed the throughput capacity of the network receivers, the receivers may drop packets forcing senders to retransmit the data after a delay. Although this will not result in any data loss, latency will increase due to the data packets re-transmissions resulting in I/O performance degradation. The flow control feature is designed to allow network receivers to slow down network senders to avoid data re-transmissions, and the delay time is much less than the overhead of re-transmitting packets. It is recommended to enable flow control on all switch ports and NIC ports that handle iSCSI traffic.

• Disable unicast storm control on switch ports. Many switches have traffic storm control

features that prevent ports from being disrupted by broadcast, multicast, or unicast traffic storms on physical interfaces. These features typically work by discarding network packages when the traffic on an interface reaches a certain percentage of the overall load. Because iSCSI traffic is unicast traffic and can typically use the entire link, it is recommended that the unicast storm control feature be disabled on switches that handle iSCSI traffic. However, the use of broadcast and multicast storm control is encouraged on all switches.

• Enable jumbo frames on switches and NICs. Jumbo frames extend the standard

Ethernet frame size to allow more data to be transferred with each Ethernet transaction. It is recommended to enable jumbo frames on switches that handle iSCSI traffic and on the NICs.


For more information on PS Series network configuration best practices and recommendations, refer to the Dell EqualLogic white paper “PS Series Groups: Storage Array Network Performance Guidelines” available at http://www.equallogic.com/psp/PDF/tr-network-guidelines-TR1017.pdf.

A high bandwidth, inter-switch link is critical to achieve optimal iSCSI SAN performance. As illustrated in Figure 3, the two Dell PowerConnect 6248 gigabit Ethernet switches are stacked with 48 Gbps inter-switch bandwidth.

Server Configuration Each Oracle 11g RAC database cluster node should be architected for optimal availability. The following sections will detail how to set up the Oracle private interconnect network and iSCSI SAN Ethernet interfaces. These are the two channels used by the database nodes to communicate with each other and to the storage system. Ensuring that these interfaces are fault tolerant will help increase the availability of the overall system.

Configuring Fully Redundant Ethernet Interconnects In addition to the iSCSI storage area network Ethernet interfaces, each Oracle 11g RAC database server needs at least three additional network interface cards (NIC) ports: one port for the external interface, and two ports for the private interconnect network. The servers in an Oracle 11g RAC are bound together using cluster management software called Oracle Clusterware that enables the servers to work together as a single entity. Servers in the cluster communicate and monitor cluster status using a dedicated private network, also known as the cluster interconnect or private interconnect. One of the servers in the RAC cluster is always designated as the master node. In a non-redundant deployment, if the private interconnect or a network switch fails and server communication to the master node is lost, then the master node will initiate a recovery of the failed database instance on a different server. This recovery is initiated to ensure that the critical data contained in the database will remain consistent and not become corrupted. The master node will then proceed to recover all of the failed instances in the cluster before providing a service from a single node that will result in a significant reduction in the level of service and available capacity. Therefore, Dell recommends that users implement a fully-redundant interconnect network configuration, with redundant private NICs on each server and redundant private network switches. Figure 4 illustrates the CAT 5E/6 Ethernet cabling of a fully-redundant interconnect network configuration of a two-node PowerEdge RAC cluster, with two private NICs on each server, and two private network switches, as shown in the red colored cables that represent the Oracle cluster private interconnect network. For this type of redundancy to operate successfully, it requires an implementation of the Link Aggregation Group, where one or more links are provided between the switches themselves. These two private interconnect network connections work independently from the public network connection. To implement a fully-redundant interconnect configuration requires the implementation of NIC teaming software at the operating system level. This software operates at the network driver level to provide two physical network interfaces to operate underneath a single IP address. For details on configuring NIC teaming, refer to “Configuring the Private NIC Teaming” in the following section.

Configuring Multiple Ethernet Interfaces for iSCSI Storage Area Networks For high availability and optimal performance, it is recommended that multiple NIC ports are used on each Oracle RAC host to communicate with the PS Series iSCSI storage arrays. A minimum of two NIC ports are required to provide redundant links to the storage arrays; in the event of a


NIC port failure in a single NIC port environment, the host will lose access to the storage until the failed NIC is physically replaced. As illustrated in Figure 4, four NIC interfaces on each of the PowerEdge servers hosting the Oracle 11g RAC database are configured to route iSCSI traffic.

Software Configuration

Operating System Configuration

Configuring Host Access to iSCSI Volumes A number of volumes are created in the PS4000XV storage array to use as the Oracle storage space. In order to access these volumes, the iSCSI initiator software needs to be installed and configured. The Open-iSCSI initiator enables the connection of a host running Red Hat Enterprise Linux 5 to an external iSCSI storage array. During the installation of the open-iSCSI initiator, ensure that the Initiator Service, the Software Initiator, and the Microsoft MPIO Multipathing Support for iSCSI installation options are all selected. On hosts running Red Hat Enterprise Linux 5, accessing these volumes requires the open-iSCSI initiator rpm iscsi-initiator-utils to be installed. Once a connection has been established between the host and the external iSCSI storage, host initiators can be created. It is a best practice to have a host initiator for each physical NIC port to ensure high availability. After creating the host initiators, bind the physical NIC port with the iSCSI host initiator using the NIC’s hardware address. The example below shows the creation of a host initiator, and binding a NIC’s hardware address to the iSCSI initiator. iscsiadm –m iface –I ieth# --op=new iscsiadm –m iface –I ieth# --op=update –n iface.hwaddress –v <MAC ADDR of eth#> Once your iSCSI interfaces have been created and attached to the appropriate NIC ports, targets can be discovered. The following command discovers the targets at a given IP address. iscsiadm –-mode discovery –-type sendtargets –-portal <ip_address> Once all your Ethernet devices have been added, the configuration of a host accessing your iSCSI volumes is complete once you log the target sessions using the following command:

iscsiadm –m node –-loginall=all

For hosts running RHEL 5, a device mapper allows them to have multiple connected paths to the same SAN volume for increased redundancy, performance, and I/O rebalancing across all active paths. To utilize the device mapper, the rpm package device-mapper-multipath must be installed. Once installed, verify that the iSCSI sessions are up and running using the iscsiadm –m session command. The installation of the rpm package will create a default /etc/multipath.conf file that will need modification. Open the /etc/multipath.conf file and comment out the line devnode “*” from the file, as this will blacklist all block devices by default. Identify your local storage ID and exclude it from your /etc/multipath.conf configuration file under the blacklist section. The SCSI ID can be found using the following command: iscsi_id –gus /block/sd# where # is the letter of your local storage.


Lastly, modify the /etc/multipath.conf template file with the appropriate information for your existing environment. It is important to remember that it is a best practice for enterprise environments to be highly available, with multiple hosts that each have multiple paths to storage and volumes. Once your multipath configuration file is set, start the multipathd service by entering the following command: service multipathd start

Configuring the Oracle Private Network NIC Teaming Dell recommends that users install two physical private NICs on each of the Oracle 11g RAC cluster servers to prevent private network communication failures. In addition to installing the two NICs, NIC teaming software is required to bond the two private network interfaces together to operate under a single IP address. The NIC teaming software provides failover functionality. If a failure occurs that impacts one of the NIC interfaces – such as a switch port failure, cable disconnection, or NIC failures – network traffic is routed to the remaining operable NIC interface. The failover is transparent to the Oracle 11g RAC database, with no network communication interruption or changes to the private IP address.

Configuring Network Requirements on All Nodes It is important that all nodes in an Oracle 11g RAC cluster have the same network interface name for the public interface. For example, if “eth0” is configured as the public interface on the first node, then “eth0” should also be selected as the public interface on all of the other nodes. This is required for the correct operation of the Virtual IP (VIP) addresses configured during the Oracle Clusterware software installation. It is important to verify the network configuration using ping tests prior to the Oracle installation. The public IP address and the private IP address must be resolvable by pinging from each node to any other nodes of the cluster. Oracle provides the Cluster Verification Utility (CVU) to verify the cluster configuration at any stage during the installation process, and also when the database is in production. The following command checks the general hardware and operating system configuration and may be used to verify network connectivity between all cluster nodes: cluvfy stage –post hwos –n node_list [-verbose]

Configuring Host Equivalence During the installation of Oracle 11g RAC software, the Oracle Universal Installer (OUI) is initiated on one of the RAC cluster nodes. OUI operates by copying files to, and running commands on, the other servers in the cluster. A secure shell (SSH) must be configured to allow OUI to accomplish these tasks, and so that no prompts or warnings are received when connecting between hosts. Use the following CVU command to check the host equivalence configuration: cluvfy stage –post hwos –n node_list [-verbose]

Configuring Shared Storage for Oracle Clusterware Using Block Device Before installing Oracle 11g RAC Clusterware software, it is necessary, at a minimum, for shared storage to be available on all cluster nodes to be used by the OCR and the Clusterware CSS Voting Disk. As shown in Table 1, the sample volume configuration includes one volume for the OCR and the Voting Disk files. Starting with Oracle 11g, the OCR and CSS Voting disk files can be placed on a shared block device.


Configuring Shared Storage for the Database Using the ASM Library Driver Two separate volumes are created for the data storage area and the Flash Recovery Area, respectively. It is recommended that these two volumes be configured as ASM disks to benefit from the ASM capabilities. For Oracle 11g R1 databases on RHEL 5 update 3, ASM requires the installation of a number of additional rpm packages including: • oracleasm-2.6.18-128.el5-2.0.5-1.el5.x86_64.rpm – the kernel module for the

ASM library that is specific to kernel 2.6.18-128.el5 • oracleasm-support-2.1.3-1.el5.x86_64.rpm – the utilities needed to administer

ASMLib • oracleasmlib-2.0.4-1.el5.x86_64.rpm – the ASM libraries ASM allows DBAs to define a storage pool called a disk group; the Oracle kernel manages the file naming and placement of the database files in that storage pool. DBAs can change the storage allocation by, adding or removing disks with SQL commands such as create diskgroup, alter diskgroup, and drop diskgroup. The disk groups can also be managed by the Oracle Enterprise Manager (OEM) and the Oracle Database Configuration Assistant (DBCA). Each Oracle 11g RAC node will contain an ASM instance that has access to the backend storage. The ASM instance, similar to database instance, communicates with other instances in the RAC environment and also features failover technology. Use the following CVU command to check the accessibility of the shared storage: cluvfy stage –post hwos –n node_list [-verbose]

Oracle 11g R1 Configuration

Installing Oracle Clusterware and Database Software The preferred method to install Oracle Clusterware the database is to use OUI. OUI provides an installation wizard to install the Clusterware and database binaries on Red Hat Enterprise Linux. During the Clusterware and Oracle installation, the OUI will ask for general information such as inventory directory paths, multi-node information, and so on. The OUI RAC deployment feature is enhanced with the ability to push the required binaries to multiple RAC nodes from one master server. The general installation order is as follows:

1. Install Oracle 11g R1 (11.1.0.6) Clusterware software. 2. Install Oracle 11g R1 (11.1.0.6) database software. 3. Upgrade Oracle Clusterware software to 11.1.0.7 patch set. 4. Upgrade Oracle database software to 11.1.0.7 patch set. 5. Create the listener and cluster database.

Installing Recommended Oracle Patches Oracle releases many patches with each database release to fix issues and enhance security features. It is critical to keep database patches up-to-date to ensure a healthy and secure environment. Check the Dell Oracle 11g deployment guide at www.dell.com/oracle for the Oracle and Dell recommended patch list.


Performance Capabilities of Oracle 11g SE RAC Test Tools and Configuration

Quest Benchmark Factory TPC-C

To understand the performance capabilities of Oracle 11g SE RAC, the Dell Oracle Solutions engineering team conducted a series of benchmark stress tests using the Benchmark Factory® for Databases from Quest software. While Benchmark Factory offers numerous industry-standard benchmarks, the team selected benchmarks similar to the TPC-C benchmark from the Transaction Processing Performance Council (TPC). This benchmark measures online transaction processing (OLTP) workloads, combining read-only and update-intensive transactions that simulate the activities found in a complex OLTP enterprise environment. The Benchmark Factory TPC-C tests conducted by the team simulated loads from 100 to 10,000 concurrent users in an increment of 100. The test outputs include metrics such as transactions per second (TPS).

The Solutions Engineering team set up the software to run on the cluster nodes as follows: • A single node Oracle 11.1.0.7 SE database: 100 to 10,000 users • Two-node Oracle 11.1.0.7 SE RAC database: 100 to 10,000 users • Three-node Oracle 11.1.0.7 SE RAC database: 100 to 10,000 users • Four-node Oracle 11.1.0.7 SE RAC database: 100 to 10,000 users

The goal of these tests was to determine:

• How many concurrent users each database may sustain • How the RAC cluster would scale as additional nodes are added

Hardware and Software Configuration

The database used in the test was running Oracle 11g (11.1.0.7) Standard Edition Real Application Cluster (RAC). The total database schema size was 160 GB, populated by Benchmark Factory. The Benchmark Factory test configuration is summarized in Table 2. Server Up to four Dell PowerEdge R610 servers with:

• A single Intel® Xeon® X5560 quad-core 2.80 GHz CPU with 8M cache, 6.40 GT/s QPI and TURBO and HT mode enabled

• 24 GB of RAM or 48 GB of RAM, with 4GB or 8GB DIMMs respectively

• Four 1Gb Broadcom NetXtreme II NIC ports for iSCSI traffic External Storage Two DellTM EqualLogicTM PS4000XV iSCSI storage arrays, each with

16 15K RPM 146GB SAS hard drives Volume Configuration Three 220 GB volumes for database files; One 150 GB volume for

Flash Recovery area; One 2 GB volume for OCR, CSS, and SPFILE OS and Device Driver Red Hat Enterprise Linux 5 Update 3

• Open iSCSI initiator iscsi-initiator-utils-6.2.0.868-0.18.el5 • Device Mapper multipath driver device-mapper-multipath-

0.4.7-23.el5 Storage Network Two stacked Dell PowerConnect 6248 gigabit Ethernet switches for

iSCSI SAN Test Software • Quest Benchmark Factory 5.7.1

• Oracle 64 bit 11.1.0.7 SE RAC Database Configuration Up to four Oracle 11.1.0.7 SE RAC with:

• 13 GB memory_target on each instance Table 2: Hardware and Software Configurations for Benchmark Factory Test


Performance Monitoring The testing utilized a number of monitoring tools to characterize cluster components. By analyzing and interpreting the statistics generated by these tools, we can identify where the performance bottlenecks occur. Three system performance statistics were sampled during the performance stress runs:

• CPU utilization • Physical memory utilization • Storage system throughput

SAR The system activity report (sar) is a Linux command to collect, report, and save system activity information. It reports system performance information, with its default CPU utilization report being the most commonly used utility to monitor CPU performance. If CPU utilization is near 100 percent, the workload sampled may be CPU-bound. The sar command uses two user specified parameters – an interval time in seconds between sampling periods, and a count parameter to set the number of sampling periods in total. The test team executed the following command to collect database server CPU performance during each benchmark factory runs: nohup sar 1 30000 > ~/sar-4nodes-run195-node1.txt & In this example, CPU performance is collected every 1 second for 30000 seconds, approximately 8 hours. The output is saved to a text file named sar-4nodes-run195-node1.txt. Figure 5 is an excerpt of the CPU performance information displayed by the command: Linux 2.6.18-128.el5 (r610-n2.us.dell.com) 06/11/2009

CPU %user %nice %system %iowait %steal %idle 02:31:43 PM all 0.38 0.00 0.25 4.25 0.00 95.12 02:31:44 PM all 0.25 0.00 0.25 4.12 0.00 95.38 02:31:45 PM all 0.12 0.00 0.50 0.25 0.00 99.13 02:31:46 PM all 0.62 0.00 0.62 1.12 0.00 97.62 02:31:47 PM all 0.87 0.00 1.12 1.99 0.00 96.01 02:31:48 PM all 0.13 0.00 0.38 5.63 0.00 93.87 02:31:49 PM all 0.62 0.00 0.37 1.87 0.00 97.13 ... 02:33:29 PM all 0.38 0.00 0.12 0.25 0.00 99.25 02:33:30 PM all 0.00 0.00 0.25 0.00 0.00 99.75 02:33:31 PM all 0.25 0.00 0.62 0.25 0.00 98.88 02:33:32 PM all 0.37 0.00 0.25 0.00 0.00 99.38 02:33:33 PM all 0.50 0.00 0.50 1.13 0.00 97.87 02:33:34 PM all 0.25 0.00 0.25 0.25 0.00 99.25

Figure 5: Sample section of the SAR report showing CPU statistics during each interval The %idle column shows the CPU idle time percentage; the system CPU utilization can be calculated as 100 - %idle. The sar reports are time stamp based. In order to interpret the impact of the Benchmark Factory stress test on CPU performance, we developed a utility to correlate the sar report’s CPU performance numbers with the Benchmark Factory user load by mapping the time stamps between the two tools. A sample of the Benchmark Factory’s “Real-Time Detail” report is shown in Figure 6. The original report did not list the time stamp column, so the Solutions Engineering team calculated it by adding the performance run start time stamp with the value in the “Elapsed Seconds” column.


Real-Time Detail Counter UserLoad Sequence Test Phase Elapsed Seconds Value Time Stamp AVG_RESP_TIME 100 33 128 168 85 2:31:38 PM AVG_RESP_TIME 100 34 128 173 69 2:31:43 PM AVG_RESP_TIME 100 35 128 178 80 2:31:48 PM AVG_RESP_TIME 100 36 128 183 158 2:31:53 PM .... AVG_RESP_TIME 200 54 32 279 39 2:33:29 PM AVG_RESP_TIME 200 55 32 284 1032 2:33:34 PM AVG_RESP_TIME 200 56 32 289 670 2:33:39 PM AVG_RESP_TIME 200 57 32 294 523 2:33:44 PM ....

Figure 6: Sample of the Benchmark Factory Real-Time Detail Report In order to generate a relationship between CPU utilization and the user load, the Solutions Engineering team wrote a series of scripts that would look up the user load and the corresponding CPU utilization at the same point in time; an example can been seen in the highlighted results in Figure 5 and Figure 6. Once data points were captured for the CPU utilization percentage (100 - %idle) at different user loads, graphs were plotted to show this relationship. OS Watcher In addition to CPU utilization, the engineering team examined the memory characteristics of each database cluster node. For simplicity, the team made the assumption that the characteristics exhibited for one node within a cluster may be generalized to the other nodes within the cluster, due to the ability of Oracle RAC to evenly balance the workload within a cluster. The memory statistics were captured using OS Watcher (OSW). OSW is a collection of Linux shell scripts used to collect and archive operating system metrics to aid in diagnosing performance issues. OSW operates as a set of background processes on the server, and gathers OS data on a regular basis. The OSW utility is started by executing the startOSW.sh shell script from the directory where OSW is installed. The script has two input parameters that control the frequency that data is collected and the number of hour’s worth of data to archive. In the following example command, OSW collects data every 5 seconds and archives the data from the past 4 days: nohup ./startOSW.sh 5 96 & One of the statistics that OSW collects is system memory usage. OSW saves the output of /proc/meminfo based on the user specified value. The sample /proc/meminfo output in Figure 7 shows a system with a total of 48 GB of RAM and has 34 GB of free physical memory. For each poll time that measured free memory, the engineering team developed a utility to cross reference the corresponding time stamp in the Benchmark Factory Real-Time Detail report (as shown in Figure 6), in order to plot memory usage versus user load. Although buffers and caches would be considered “free” memory, those data points were not charted within the graphs. zzz ***Sun Jun 21 23:00:04 CDT 2009 MemTotal: 49434036 kB MemFree: 34270184 kB Buffers: 353068 kB Cached: 12631124 kB SwapCached: 7732 kB Active: 10277800 kB Inactive: 4483064 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 49434036 kB LowFree: 34270184 kB SwapTotal: 8388600 kB


SwapFree: 8239712 kB … zzz ***Sun Jun 21 23:04:59 CDT 2009 MemTotal: 49434036 kB MemFree: 34270564 kB Buffers: 354892 kB Cached: 12637136 kB SwapCached: 7732 kB Active: 10278772 kB Inactive: 4483804 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 49434036 kB LowFree: 34270564 kB SwapTotal: 8388600 kB SwapFree: 8239712 kB

Figure 7: Sample Section of OSW Memory Report EqualLogic SAN HeadQuarters The Dell EqualLogic SAN HeadQuarters (SAN HQ) is a centralized event and performance monitoring tool for Dell EqualLogic storage systems. SAN HQ enables administrators to monitor multiple EqualLogic storage groups from a single graphical interface. It gathers and formats performance data and other vital group information, such as I/O performance and capacity, and so on. Figure 8 displays a screen shot of SAN HQ I/O performance view.

Figure 8: Dell EqualLogic SAN HeadQuarters I/O Performance View To better understand current or past SAN attribute characteristics, administrators can use a simple slide bar in SAN HQ to select and view a long or short time window. SAN HQ also can export data sets into a .csv file for analysis and reporting. The test team used SAN HQ to collect storage I/O performance data in I/O per second (IOPS) and disk latency data points. This data was used to understand the EqualLogic PS4000XV performance during the Benchmark Factory stress tests.


Tuning Oracle Memory Size On the test database baseline Benchmark Factory tests were conducted to optimize database settings including the size of Oracle SGA memory and PGA memory. A larger SGA setting helps reduce the amount of physical I/O. However, the relationship is not linear. The optimal size of Oracle SGA can get to the point where the marginal benefit begins to decline. For our test configuration, the Oracle AWR (Automatic Workload Repository) report obtained from baseline testing shown in Figure 9 indicated that the optimal SGA size is in the range of 9 - 12 GB. The reduction in physical reads is minimized when SGA is larger than 12 GB. SGA Target Advisory SGA Target Size (M) SGA Size Factor Est DB Time (s) Est Physical Reads

3,168 0.38 1,327,870 37,779,644

4,224 0.50 895,776 20,459,187

5,280 0.63 728,220 13,742,069

6,336 0.75 641,710 10,272,675

7,392 0.88 598,910 8,558,231

8,448 1.00 569,142 7,364,453

9,504 1.13 549,509 6,576,457

10,560 1.25 534,597 5,979,936

11,616 1.38 524,011 5,555,743

12,672 1.50 519,685 5,381,942

13,728 1.63 516,046 5,236,863

14,784 1.75 516,112 5,236,863

15,840 1.88 516,117 5,236,863

16,896 2.00 516,119 5,236,863

Figure 9: Baseline Tests AWR Report Section on SGA Advisory Similarly, the AWR report revealed that the optimal PGA setting should be set when the estimated PGA over-allocation count is 0. The AWR report obtained from baseline testing shown in Figure 10 indicated that the optimal PGA size is roughly 4 GB. PGA Memory Advisory

• When using Auto Memory Mgmt, minimally choose a pga_aggregate_target value where Estd PGA Overalloc Count is 0

PGA Target Est (MB)

Size Factr

W/A MB Processed

Estd Extra W/A MB Read/ Written to Disk

Estd PGA Cache Hit %

Estd PGA Overalloc Count

Estd Time

912 0.25 564.98 15.25 97.00 113 1,740

1,824 0.50 564.98 15.25 97.00 77 1,740

2,736 0.75 564.98 15.25 97.00 50 1,740

3,648 1.00 564.98 0.00 100.00 17 1,694

4,377 1.20 564.98 0.00 100.00 0 1,694

5,107 1.40 564.98 0.00 100.00 0 1,694

5,836 1.60 564.98 0.00 100.00 0 1,694

6,566 1.80 564.98 0.00 100.00 0 1,694

7,296 2.00 564.98 0.00 100.00 0 1,694

Figure 10: Baseline Tests AWR report section on PGA Advisory Beginning with Oracle 11g, the database can manage SGA and PGA memory automatically. A new database parameter named memory_target was introduced in Oracle 11g to designate the total memory size to be used by the instance. This capability is referred to as Automatic Memory


Management. The size of the memory_target should be the total size of SGA and PGA area. The test team set the memory_target parameter to 13 GB, which is the sum of the optimal PGA and SGA size suggested by the AWR report.

Testing RAC Scalability with 24 GB RAM Per Node

CPU Performance Analysis In the scalability test, the test team started with one node then scaled the Oracle RAC cluster up to four nodes. As a consequence, there can be up to four instances of SAR data for each cluster depicting the CPU utilization for each of the four nodes. The graphs below depict only one selected server’s SAR report that shows CPU utilization for one node; a single node is shown, but is representative of the others due to the ability of Oracle RAC to evenly distribute workloads in the cluster. As a starting point, each database server has 24 GB of RAM installed; figures 11, 12, and 13 show the CPU utilization versus the user load for each node with 24 GB of physical memory.

Figure 11: CPU Utilization vs. User Load - 2-node RAC with 24 GB Memory per Node

0

20

40

60

80

100

120

100

200

300

400

500

500

600

700

800

900

1000

1100

1100

1200

1300

1400

1500

1600

1700

1700

1800

1900

2000

2100

2200

2300

2300

2400

2500

CPU

uti

lizat

ion

(%)

User Load

CPU utilization of a node within a 2-node RAC cluster with 24 GB RAM per node

2-node RAC, 24GB RAM 40 periods moving average


Figure 12: CPU Utilization vs. User Load - 3-node RAC with 24 GB Memory per Node

Figure 13: CPU Utilization vs. User Load - 4-node RAC with 24 GB Memory per Node In each of the figures above, the black line indicates the moving average CPU utilization percentage. A few key general trends were observed:

• As we add each node to the Oracle RAC cluster, not only do we scale out with the total number of users supported by the RAC cluster, but the moving average for CPU utilization per node also goes down. For example, with two nodes in the cluster, the moving average CPU utilization for a 2400 user load is approximately 50% (Figure 11), and as we add one more node to the cluster making it a 3 node cluster, the moving average CPU utilization for the same user load is down to 30% (Figure 12). We further observe this trend with the graph in Figure 13 that shows the moving average CPU utilization reduces further to approximately 20% for the same 2400 user load when we add another node to a 3 node RAC cluster.

0

20

40

60

80

100

120

2300

2400

2400

2400

2500

2500

2500

2600

2600

2600

2700

2700

2700

2800

2800

2800

2800

2900

2900

2900

3000

3000

3000

3100

3100

3100

3200

3200

3200

CPU

uti

lizat

ion

(%)

User Load



0

20

40

60

80

100

120

300

400

500

700

800

1000

1100

1300

1400

1500

1700

1800

2000

2100

2300

2400

2500

2700

2800

3000

3100

3300

3400

3500

3700

3800

4000

4100

4200

CPU

uti

lizat

ion

(%)

User Load




• The overall CPU utilization still had room for more growth. For example, when Oracle was not able to handle any more user connections due to other bottlenecks, for each of the RAC clusters the CPU utilization was around 60-65%.

Thus the graphs indicate that even when scaled up to four single socket nodes with 24 GB of memory per node, the total number of users that the configuration can support was not limited by the CPU.

Memory Utilization Analysis Figure 14 displays the memory utilization as it relates to increasing user loads. The red line represents the free memory available on one node as the user load increases in a 3-node RAC cluster, while the blue line represents the same information in a 4-node RAC cluster. Each of these clusters has 24 GB of physical memory per node. In both cases, the graph was generated using data from a single node within an N node cluster. It may be assumed that these lines are representative of the other cluster nodes because of the ability of Oracle RAC to evenly distribute workloads.

Figure 14: Memory Utilization vs. User Load – 3, 4-Node RAC with 24 GB RAM per Node For a three node cluster, the available physical memory is approximately 125 MB during runs with 2300 – 3200 users; during this run the logs show swap usage. When the system is scaled to 4 nodes, represented by the blue line, at approximately 3900 users the system started to level out with 125 MB of available physical memory. Until the user load reaches 4300 we see the use of swap, due to insufficient physical memory to handle the additional transactions from Benchmark Factory. The operating system can accommodate these increasing transactions only by swapping out some existing pages in physical memory to the virtual memory or swap disk; this is represented by the flattening lines in the figure above.

Storage System Performance Analysis Since two Dell EqualLogic PS4000 Series arrays can be allowed in a PS Series group, our test configuration was limited to two members. The Dell EqualLogic SAN HeadQuarters (SAN HQ) performance monitoring tool was used to capture the IOPS and disk latencies. IOPS data was extracted from the compressed output generated by SAN HQ during the stress tests. These

0.1

1

10

100

100

300

500

700

1000

1200

1400

1600

1900

2100

2300

2600

2800

3000

3200

3500

3700

3900

4100

Free

mem

ory

(GB)

on

each

RA

C no

de

User Load

Memory utilization of RAC node with 24 GB RAM per node

4 node RAC, 24 GB RAM


~ 125 MB free memory per node


extracted data contained the average IOPS and disk latency data points in 30-minute time intervals. SAN HQ allows you to extract these data points from the monitoring tool into an Excel spread sheet that was used to plot Figure 15 below. All of the IOPS data points were well within the acceptable disk latency range of 20 ms or less.

Figure 15: IOPS vs. # of RAC Nodes - PS4000XV with 24 GB RAM per Node Figure 15 shows the member array IOPS performance during the scalability tests, ranging from a single node to a 4-node Oracle RAC cluster. As shown in the figure, with an increasing number of RAC nodes with 24 GB of physical memory, the two storage members were able to scale steadily and reach a maximum of 1800 IOPS per member. This IOPS value is well within the maximum IOPS, and acceptable disk latency, that a single Dell EqualLogic PS4000XV array can deliver. The workload is equally balanced between the PS4000XV members, as the database volume data blocks are evenly distributed between the two members.

Examining both CPU and Memory Utilization If we examine CPU usage in Figure 12, when the 3-node cluster begins to swap (Figure 14), CPU utilization at a user load of 2400 was approximately 28%. As Benchmark Factory continued to increase the user load, the swapping continues as the system is forced to swap to disk because of the lack of free physical memory. The stress test starts to fail once the user load reaches 3200, and at this point we note that the CPU utilization was approximately 42%. Similar results (Figure 13 and 14) can be observed with the 4-node cluster, where we can see that the system starts to have low physical memory availability when the user load reaches 3800 with CPU utilization approximately around 40%. The test starts to fail once the user load reaches 4300, and at this point we note that CPU utilization is approximately 50%. During the swap time we can continue to load the database, but it’s likely in a yellow zone because as we keep increasing the user loads the database servers start to get unstable due to a low amount of free memory available.

0

500

1000

1500

2000

2500

3000

3500

1 2 3 4

PS40

00XV

arr

ay IO

Per

Sec

ond

Number of Oracle RAC nodes

PS4000XV IOPS with 24 GB RAM in each RAC node

member1 24GB RAM

member2 24GB RAM


Overall Performance Analysis Figure 16 shows the final Benchmark Factory results for all the nodes with 24 GB of physical memory per node. This result shows that while the RAC scaled as nodes were added, the scalability is less than linear.

Figure 16: TPS vs. User Load - 1 to 4 RAC Nodes with 24 GB RAM per Node From all the above critical system’s performance metrics, there are few common trends:

• As the systems near their maximum memory performance, they progressively utilize more swap space and the database starts to get unstable.

• When comparing the CPU graph with the memory graph, at the maximum user load there is still ample CPU capability even once the test fails.

• Given that the Oracle memory area (memory_target) is set to 13 GB to minimize physical I/O, even at the maximum 4-node cluster size the storage members were delivering IOPS well below their potential, suggesting that the storage disks were not the bottleneck for the overall performance.

Therefore, we concluded that due to low available free physical memory, aggressive swapping, and low CPU utilization at failure time, the clusters are not balanced in respects to memory to CPU utilization ratios; this system’s performance was limited by memory. In order to confirm this conclusion, the next step in our testing was to increase the memory from 24 GB to 48 GB per node, in other words to increase the memory in a PowerEdge R610 server with 1 CPU socket to the maximum.

Testing RAC Salability with 48 GB RAM Per Node For the next testing phase, we scaled-up our clusters and upgraded each node in the cluster to 48 GB of physical memory while keeping rest of the configuration the same. As of the authoring of this paper, 48 GB (8 GB RDIMMs * 6 Memory Banks) is the maximum memory capacity that a Dell PowerEdge R610 can support per processor socket. As part of our scalability study, since each cluster node has only one processor socket populated, all 48 GB of memory DIMMs must be placed in the banks adjacent to the populated processor due to the NUMA architecture of the server system. Although the R610 server has additional free memory banks available next to the other processor socket, these banks cannot be used as the socket must be populated.

0

50

100

150

200

250

100

300

500

700

900

1100

1300

1500

1700

1900

2100

2300

2500

2700

2900

3100

3300

3500

3700

3900

4100

4300

Tran

sact

ion

Per

Seco

nd

User Load

Benchmark Factory results for 1 to 4 RAC nodes with 24 GB RAM per node

4 nodes

3 nodes

2 nodes

1 node


For the 48 GB cluster, Benchmark Factory ran again on one, two, three, and finally four nodes within the cluster and an incremental user load of 100 on the database was applied until the database failed. The same tool sets and scripts that were used for the 24 GB cluster were again used to correlate CPU utilization and memory utilization, with the corresponding user load for each of the nodes within the cluster. The following sections and graphs present the different performance metrics captured, and the analysis for this configuration.

CPU Performance Analysis With each node configured with 48 GB of physical memory, Figures 17-20 chart CPU utilization on a point for point basis in relationship to the user load at a particular common time stamp with the black line representing a 30 point moving average. Similar to the 24 GB cluster’s CPU performance characteristics, observation and comparison of the 4 graphs demonstrate that for a given user load CPU utilization continue to decrease as more nodes are added to the cluster. For example, at a 2300 user load the CPU utilization for the 1, 2, 3 and 4 node clusters are approximately 60-80%, 20-40%, 20-35% and 10-25% respectively.

Figure 17: CPU Utilization vs. User Load - 1-Node RAC with 48 GB RAM per Node

0

20

40

60

80

100

120

200

200

300

400

500

500

600

700

800

900

900

1000

1100

1200

1200

1300

1400

1500

1600

1600

1700

1800

1900

1900

2000

2100

2200

2300

2300

CPU

uti

lizat

ion

(%)

User Load






0

20

40

60

80

100

120

2300

2400

2500

2500

2600

2700

2800

2800

2900

3000

3000

3100

3200

3300

3300

3400

3500

3500

3600

3700

3700

3800

3900

4000

4000

4100

4200

4200

4300

CPU

uti

lizat

ion

(%)

User Load



0

20

40

60

80

100

120

300

500

700

800

1000

1200

1400

1600

1700

1900

2100

2300

2500

2600

2800

3000

3200

3400

3500

3700

3900

4100

4300

4400

4600

4800

5000

5100

5300

CPU

uti

lizat

ion

(%)

User Load




Figure 20: CPU Utilization vs. User Load - 4-Node RAC with 48 GB RAM per Node Comparing the 24 GB cluster’s CPU utilization graphs with the 48 GB cluster’s CPU utilization graphs, we can see that the CPU was utilized much more in the 48 GB cluster that resulted in a significantly higher user load that could be sustained by the clusters. For example, the 3-node cluster with 24 GB of RAM per node was able to support only up to 3200 users while the 3-node cluster with 48 GB of RAM per node was able to support up to 6200 users. Similarly, the 4-node 24 GB cluster was able to support up to 4300 users while the 4-node 48 GB cluster was able to support up to 8100 users. Given that the only thing that was changed between the two configurations was doubling the physical memory, this suggests that the CPUs were underutilized in the 24 GB configuration with memory being the issue.

Memory Utilization Analysis Figure 21 shows that eventually the free physical memory on the system will run out. The 48 GB clusters for both three and four nodes eventually leveled out to approximately 250 MB of free memory available, at which point some swap memory was utilized. In all cases, the database eventually failed as higher and higher user loads were applied.

0

20

40

60

80

100

120

200

400

600

900

1100

1400

1600

1800

2100

2300

2600

2800

3000

3300

3500

3800

4000

4300

4500

4700

5000

5200

5500

6700

6900

7200

7400

7600

7800

CPU

uti

lizat

ion

(%)

User Load




Figure 21: Memory Utilization vs. User Load - 3, 4-Node RAC with 48 GB RAM per Node

Examining both CPU and Memory Utilization It is useful to examine the database characteristics during the failure to understand why the system failed. For example for the three node cluster, the database failed at 6300 users and the four node cluster failed at 8200 users. Once the database failed, the Oracle alert.log had the following message: GES: System Load is HIGH This can be confirmed in Figure 19 and Figure 20 that shows CPU usage in the 70-100% range during this time period. As displayed by the CPU graphs, the frequency of time periods when CPU utilization was near 100% occurred at a much higher rate leading to database failures. The corresponding memory graphs show that memory usage during these peak user loads were limited and were at or under 250 MB (Figure 21). Since the CPU idle time was low, and free memory was also low for the three node cluster at 6300 users, the database may have failed due to a lack of memory or an overloaded CPU. Likewise, for the four node cluster the free memory was at approximately 250 MB, and there were frequent 100% CPU utilization spikes. Database failure may have occurred due to either a lack of memory, or high CPU utilization, or a combination of both.

Storage System Performance Analysis Figure 22 shows the IOPS performance of each of the two PS4000XV arrays during the scalability tests ranging from a single node to a 4-node Oracle RAC cluster, with 48 GB of physical memory on each node. As the physical memory increased from 24 GB to 48 GB, the system performance was no longer limited by memory, thus the database was able to handle more transactions at higher user loads. Consequently, the PS4000XV members processed more I/O requests and showed much improved IOPS results comparing to the 24 GB configuration as show in Figure 23. The corresponding disk latency performance was still within the acceptable range of 20 ms in all test durations of the 48 GB memory configurations, except towards the end of run on the 4-node RAC, when disk latency started to have a delayed response of above 20 ms. This indicates that the PS4000XV showed signs of stress after approximately 7000 user loads on the 4-node RAC.

0.1

1

10

100

200

600

1000

1400

1800

2200

2600

3000

3400

3800

4200

4600

5000

5400

5800

6300

6700

7100

7500

7700

Free

mem

ory

(GB)

on

each

RA

C no

de

User Load

RAC node memory utilization with 48 GB RAM per node


4 node RAC, 48 GB RAM~ 250 MB free memory per node


Figure 22: IOPS vs. # RAC Nodes - PS4000XV with 48 GB RAM per Node

Figure 23: IOPS vs. # RAC Nodes - PS4000XV Comparing 24 GB to 48 GB

Overall Performance Analysis Figure 24 displays the final Benchmark Factory results for all the nodes with 48 GB of physical memory per node and shows that RAC scaled as nodes were added.

0

500

1000

1500

2000

2500

3000

3500

1 2 3 4

PS40

00XV

arr

ay IO

Per

Sec

ond


PS4000XV IOPS with 48 GB RAM in each node

member1 48GB RAM

member2 48GB RAM

0

500

1000

1500

2000

2500

3000

3500

1 2 3 4

PS40

00XV

arr

ay IO

Per

Sec

ond


PS4000XV IOPS vs. number of RAC nodes

member1 24GB RAM

member2 24GB RAM

member1 48GB RAM

member2 48GB RAM


Figure 24: TPS vs. User Load - 1 to 4-Node RAC with 48 GB RAM per Node As illustrated in Figure 25, the 48 GB Oracle RAC cluster scales better than its 24 GB counterpart. For the 24 GB configuration, the TPS measure scales in the range of 40% - 60% with the addition of each node. In such a configuration, memory became the bottleneck before any other systems resources were significantly utilized. By adding more system memory to 48 GB per node, the TPS measure improved. The results demonstrate a near linear scalability, in the range of 75%-95% and that the benefit of additional memory became more apparent with each additional node.

Figure 25: Oracle 11g SE RAC Scalability with 24 GB RAM or 48 GB RAM per Node

Performance Characteristics Summary

0

50

100

150

200

250

300

350

400

450

100

500

900

1300

1700

2100

2500

2900

3300

3700

4100

4500

4900

5300

5700

6100

6500

6900

7300

7700

8100

Tran

sact

ion

Per

Seco

nd

User Load

Benchmark Factory results for 1 to 4 RAC nodes with 48 GB RAM per node

4 nodes

3 nodes

2 nodes

1 node

0

50

100

150

200

250

300

350

400

450

1 2 3 4

Tran

sact

ion

Per

Seco

nd

Number of nodes in the 11g RAC cluster

Oracle 11g SE RAC scalability

24 GB RAM

48 GB RAM


In the RAC cluster with 48 GB of physical memory per node, the test team noticed that both the CPU and memory resources become limited almost simultaneously. In an ideal situation, all the resources would be constrained simultaneously. Extra CPU capability without complementary memory may not lead to ideal scaling characteristics, and likewise extra memory without CPU capability may be unnecessary. By adequately balancing memory, CPU and storage you can help make the most out of scaling out an Oracle RAC cluster. A bottleneck in any one component such as storage, CPU, or memory may result in non-optimal scaling. Likewise, scaling up of one component when another component is the bottleneck for the system may not add performance and may be unnecessary. From this scalability study where we were able to stress the cluster resources such as the CPU, memory, and the storage throughput to its near maximum capability, we may conclude that for a typical OLTP environment similar to what was used in our test configuration, with similarly sized database, a 4-node Dell PowerEdge R610’s with Oracle 11g SE RAC and two PS4000XV storage arrays, it may support up to 6000 users and deliver near 300 transactions per second; system resources are utilized at a percentage comparable to a real world environment. Furthermore, from the high availability perspective, in the case of a node failure in a 4-node cluster, the workload may continue be supported in the three surviving nodes with adequate system resources. Since the maximum resources utilized in this scalability study was 4 CPUs across 4-nodes, with 4-cores each when this paper was authored, is limited by the Oracle SE database RAC license, to achieve better performance would require an upgrade to Oracle’s EE database. Oracle EE would allow for more nodes per cluster, that in turn allows for more CPUs and more physical memory that leads to improved performance. Also since the team observed that the 4-node Oracle SE RAC was able to fully utilize the two PS4000XV storage arrays, with limitation of only two PS4000’s per storage group in order to continue to scale out with Oracle EE the PS4000 arrays would have to be updated to a higher-end storage system such as the Dell EqualLogic PS6000’s.


Solution Deliverables List for Oracle 11g on Dell EqualLogic PS4000XV iSCSI Storage This section contains the solution deliverables List (SDL) for the Oracle 11g on Dell EqualLogic PS4000XV iSCSI storage arrays. It contains a detailed list of server and storage hardware configurations, firmware, driver, operating system, and database versions.

Recommended Hardware/Software Requirements (For details, see below)

Validated Component(s) Minimum Oracle RAC Configuration

PowerEdge Nodes PowerEdge R610 1

Memory All valid PowerEdge R610 memory configurations 1 GB (per node)

PS Series Storage Array PS4000XV 1

Ethernet Ports Intel or Broadcom gigabit NICs 5 Ethernet Switches for Oracle Private Interconnect Network

Any Dell PowerConnect gigabit-only Switches 2

Ethernet Switches for iSCSI Storage Area Network

Dell PowerConnect 62xx gigabit Switches 2

Raid Controllers (Used for internal storage only)

PERC 6/i 1 (Per Node)

Internal Drive All valid PowerEdge R610 internal storage configurations 73 GB/node

Oracle Software & Licenses

Oracle 11g R1 11.1.0.6 SE (Base) + Oracle patch set 11.1.0.7 RAC

Operating System Enterprise Linux 5 Update 3 Table 3: Minimal Hardware/Software Requirements


Validated Servers

PowerEdge Servers

Model BIOS [*] iDRAC Firmware [*] Notes

R610 1.1.4 1.0.3

Internal Disks RAID PERC 6/i Firmware version = 6.1.1-0047; Driver version =

00.00.04.01-RH1 SAS/SATA Backplane 0:0 Backplane Firmware version = v1.07 A00;

Network Interconnect Intel PRO/1000 Network Drivers Driver version = 0.5.11.2-NAPI Broadcom NetXtreme II Gigabit Ethernet Driver Driver version = bnx2 v1.8.7b

Ethernet Teaming Driver Driver version = v3.2.4 iSCSI SAN Switches

Dell PowerConnect 6248 Gigabit Ethernet Switches Firmware = 2.2.0.3

iSCSI Storage Dell EqualLogic iSCSI Storage PS4000XV; Firmware = 4.1.4

Software Oracle 11g R1 11.1.0.7 Standard Edition x86_64 Operating systems Enterprise Linux 5 Update 3

iSCSI Initiator iscsi-initiator-utils-6.2.0.868-0.18.el5

Device Mapper multipath driver device-mapper-multipath-0.4.7-23.el5 Table 4 : Detailed Firmware, Driver and Software Versions NOTES: *: Minimum BIOS and iDRAC versions. For the latest BIOS updates see http://support.dell.com


Conclusion Dell solutions for Oracle 11g SE database are designed to simplify operations, improve usage, and cost-effectively scale as your needs grow over time. This reference configuration white paper provides a blueprint for setting up an Oracle 11g SE RAC database on Dell EqualLogic iSCSI storage arrays and Dell PowerEdge servers. The best practices described in this paper are intended to help achieve optimal performance of Oracle 11g on Enterprise Linux 5 Update 3. To learn more about deploying Oracle 11g RAC on Dell storage systems and PowerEdge servers, visit www.dell.com/oracle or contact your Dell representative for the most current information on Dell servers, storage, and services for Oracle 11g solutions.


Tables and Figures Table 1: Volumes for the Oracle RAC Configuration ..................................................................... 13 Table 2: Hardware and Software Configurations for Benchmark Factory Test ............................. 19 Table 3: Minimal Hardware/Software Requirements ..................................................................... 36 Table 4 : Detailed Firmware, Driver and Software Versions ......................................................... 37 Figure 1: Oracle 11g SE on Dell EqualLogic PS4000XV iSCSI Storage Architecture Overview .... 9 Figure 2: "Pay-as-you-grow" Methodology to Scale up to Four Single Socket Servers ................ 10 Figure 3: Recommended Dual Control Module Network Configuration ........................................ 10 Figure 4: Cabling Two-Node Oracle Database 11g Standard Edition Real Application Clusters Database ....................................................................................................................................... 11 Figure 5: Sample section of the SAR report showing CPU statistics during each interval ........... 20 Figure 6: Sample of the Benchmark Factory Real-Time Detail Report ......................................... 21 Figure 7: Sample Section of OSW Memory Report ....................................................................... 22 Figure 8: Dell EqualLogic SAN HeadQuarters I/O Performance View .......................................... 22 Figure 9: Baseline Tests AWR Report Section on SGA Advisory ................................................. 23 Figure 10: Baseline Tests AWR report section on PGA Advisory ................................................. 23 Figure 11: CPU Utilization vs. User Load - 2-node RAC with 24 GB Memory per Node .............. 24 Figure 12: CPU Utilization vs. User Load - 3-node RAC with 24 GB Memory per Node .............. 25 Figure 13: CPU Utilization vs. User Load - 4-node RAC with 24 GB Memory per Node .............. 25 Figure 14: Memory Utilization vs. User Load – 3, 4-Node RAC with 24 GB RAM per Node ........ 26 Figure 15: IOPS vs. # of RAC Nodes - PS4000XV with 24 GB RAM per Node ........................... 27 Figure 16: TPS vs. User Load - 1 to 4 RAC Nodes with 24 GB RAM per Node ........................... 28 Figure 17: CPU Utilization vs. User Load - 1-Node RAC with 48 GB RAM per Node .................. 29 Figure 18: CPU Utilization vs. User Load - 2-Node RAC with 48 GB RAM per Node .................. 30 Figure 19: CPU Utilization vs. User Load - 3-Node RAC with 48 GB RAM per Node .................. 30 Figure 20: CPU Utilization vs. User Load - 4-Node RAC with 48 GB RAM per Node .................. 31 Figure 21: Memory Utilization vs. User Load - 3, 4-Node RAC with 48 GB RAM per Node ......... 32 Figure 22: IOPS vs. # RAC Nodes - PS4000XV with 48 GB RAM per Node ............................... 33 Figure 23: IOPS vs. # RAC Nodes - PS4000XV Comparing 24 GB to 48 GB .............................. 33 Figure 24: TPS vs. User Load - 1 to 4-Node RAC with 48 GB RAM per Node ............................ 34 Figure 25: Oracle 11g SE RAC Scalability with 24 GB RAM or 48 GB RAM per Node ................ 34

References 1. “PS Series Groups: Storage Array Network Performance Guidelines”, a Dell EqualLogic white

paper. http://www.equallogic.com/psp/PDF/tr-network-guidelines-TR1017.pdf

2. “Pro Oracle Database 10g RAC on Linux”, Julian Dyke and Steve Shaw, Apress, 2006. 3. “Oracle Database Licensing Information”, 11g Release 1 (11.1), B28287-17.

http://download.oracle.com/docs/cd/B28359_01/license.111/b28287.pdf

maximizing performance on a - · pdf filereference configuration for oracle 11g se rac on dell...

Documents