high performance solution for oracle 11g rac with ... · high performance solution for oracle 11g...

White Paper

EMC Solutions

Abstract

This solution demonstrates the benefits of deploying EMC® XtremCache™ and VMAX® to increase IOPS and decrease latency for OLTP databases. It provides scalability, high performance, and ease of use for mission-critical business demands.

December 2013

HIGH PERFORMANCE SOLUTION FOR ORACLE 11g RAC WITH XTREMCACHE 2.0.1 ON VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache, Red Hat Enterprise Linux, Oracle Database Enterprise Edition

Optimum IOPS for OLTP workloads

Sustained Average Response Time Less Than One Millisecond

High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,

Red Hat Enterprise Linux, Oracle Database Enterprise Edition

2

Copyright © 2013 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

The information in this publication is provided as is. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com.

All trademarks used herein are the property of their respective owners.

Part Number H11828.2

3 High Performance Solution for Oracle 11g with XtremSW 2.0.1 on VMAX Enabled by EMC VMAX 40K, XtremSF, XtremCache,


Table of contents

Executive summary ............................................................................................................................... 5

The business challenge .................................................................................................................... 5

The technology solution: open, best-in-class components ............................................................... 5

Open standards benefits ............................................................................................................. 6

Operational advantages .............................................................................................................. 6

Solution overview ............................................................................................................................ 6

Key results and recommendations ................................................................................................... 6

Introduction.......................................................................................................................................... 8

Purpose ........................................................................................................................................... 8

Scope .............................................................................................................................................. 8

Audience ......................................................................................................................................... 8

Terminology ..................................................................................................................................... 9

Technology overview .......................................................................................................................... 10

Architecture layers ......................................................................................................................... 10

Architecture diagram ...................................................................................................................... 10

EMC Oracle Performance Solution: Server layer ................................................................................. 11

Server hardware ............................................................................................................................. 12

Server software .............................................................................................................................. 13

Red Hat Enterprise Linux ................................................................................................................ 13

Oracle Grid Infrastructure and Database 11g Release 2 .................................................................. 14

XtremSF and XtremCache deployment ............................................................................................ 14

XtremSF and XtremCache overview ................................................................................................ 14

EMC XtremCache configuration .................................................................................................. 15

EMC Oracle Performance Solution: Network layer .............................................................................. 16

EMC Oracle Performance Solution: Storage layer ............................................................................... 18

Introduction ................................................................................................................................... 18

Storage components ...................................................................................................................... 18

Storage hardware ........................................................................................................................... 18

EMC Symmetrix VMAX 40K eight-engine configuration .................................................................. 19

Storage software ............................................................................................................................ 20

EMC Oracle Performance Solution: Oracle Database layer .................................................................. 21

Introduction ................................................................................................................................... 21

Storage virtual provisioning design ................................................................................................ 21

Drive type....................................................................................................................................... 21



4

ASM disk group configuration for OLTP database ........................................................................... 21

OLTP database and workload profile .............................................................................................. 22

EMC Oracle Performance Solution: Performance testing .................................................................... 23

Overview ........................................................................................................................................ 23

Test objective ................................................................................................................................. 24

Query only test scenarios and methodology ................................................................................... 24

Read test results ............................................................................................................................ 24

Update only test scenarios and methodology ................................................................................. 26

Peak write test results .................................................................................................................... 27

Conclusion ......................................................................................................................................... 29

Summary ....................................................................................................................................... 29

Findings ......................................................................................................................................... 29

References.......................................................................................................................................... 30

EMC documentation ....................................................................................................................... 30

Oracle documentation.................................................................................................................... 30



Executive summary

Customers require an open, scalable, tiered, highly available, and high performance infrastructure to run their critical Oracle systems. IT organizations must strive for better performance and increased efficiency in their Oracle infrastructure and Oracle database and storage administration operations, including the following requirements:

Reduce capital expenditures and operational expenditures by deploying an open, non lock-in technology

Consolidate many Oracle databases (Oracle database versions 10gR1 to 11gR2 and Oracle 12c) and database workloads, including OLTP and Data Warehouse, to maximize the efficiency of the data center infrastructure

Deliver maximum performance while effectively utilizing the existing arrays and Oracle infrastructure

Maintain the highest performance levels and provide predictable performance to deliver the quality of service required in these Oracle mixed workload environments

The EMC Proven High Performance Solution for Oracle Real Application Clusters (RAC) on EMC Symmetrix® VMAX® is an open architecture that incorporates open, best-in-class Intel servers with EMC server-side flash storage (XtremSF™) and EMC VMAX storage arrays.

The solution uses optimal servers to balance performance, scalability, and Oracle license costs. The use of Xtrem™ technologies XtremSF and XtremCache™ software in the servers provides distinct performance and operational advantages over equivalent systems that do not contain server-side flash technologies.

XtremCache does the following to accelerate an Oracle RAC environment:

Features an ultra-performance tier: Accelerates any application that benefits from low-latency, high bandwidth physical read I/O

Hottest data resides on database server flash

Data is as close to the Oracle Database server CPU as any storage model will allow

Cooperates with Oracle Clusterware: Oracle Clusterware is the final authority on all node membership information in an Oracle RAC deployment

Has no awareness of database instances:

Has no concern for content of blocks of cached LUNs

Only XtremCache nodes can access LUNs cached by XtremCache

Does not impose a performance penalty on active transactions for cache insertions or cache coherency

The business challenge

The technology solution: open, best-in-class components



6

Offers optimized performance: VMAX arrays are relieved of read IOPS, leaving more bandwidth for handling writes. VMAX FAST enables automatic data placement as data goes from hot to cool usage.

Delivers the highest performance levels in the industry: This solution delivers the highest performance for mixed workload Oracle environments. EMC Proven Solutions for Oracle have demonstrated sustained metrics over 3.7 million IOPS with latency of less than half a millisecond for OLTP workload, and Data Warehouse workloads with sustained throughput of 32 GBs with a data load rate of 21 TB/hour. Key results and recommendations on page 7 provides details.

Open standards benefits

This solution is based on open standards. Advantages resulting from the open standards commitment include the following:

Flexible adaptation to existing and future customer needs and open industry standards

Lower capital investment and operational expense without vendor lock-in

Operational advantages

EMC open architecture not only supports different releases of Oracle Database software (10g, 11g, and 12c), but it also provides the capability of running the databases concurrently; that is, it supports database consolidation.

Open architecture and flexible adaptation means that application modification is not required for database deployment for this solution, lessening the potential for unforeseen impact to business operations and systemic data flow throughout the enterprise.

The purpose of the solution is to build an EMC High Performance Solution for Oracle on VMAX infrastructure, based on an open architecture, and demonstrate the following capabilities of the infrastructure:

High performance and flexibility

Low operational costs

Reduced risk

This white paper validates the performance of the solution and provides guidelines to build similar solutions. This document is not a comprehensive guide to every aspect of this solution.

The core advantages of the EMC Proven High Performance Solution for Oracle RAC on VMAX are as follows:

Delivers the highest performance for Oracle RAC workload environments. EMC Proven High Performance Solution for Oracle RAC on VMAX has demonstrated sustained metrics over 3.7 million IOPS with latency of less than ½ millisecond for OLTP workload. This impressive performance is achieved by utilizing open

Solution overview

Key results and recommendations



best-in-class components at the computer, network and storage layers. Details are listed in Table 1:

Table 1. IOPS test results with workload when XtremCache is enabled

Workload type

Performance statistics

One node

Two nodes

Four nodes

Eight nodes

Read only workload

IOPS 457,136 962,155 1,914,963 3,765,176

Response time (ms) 0.74 0.69 0.68 0.75

UPDATE transaction workload

Aggregate IOPS 53,492 99,649 190,809 303,330

Write IOPS 26,950 50,110 95,932 153,276

Read IOPS 26,542 49,539 94,877 150,054

Redo throughput (MB/s)

20 37 71 115

Note: Because EMC XtremCache is write through cache, for the UPDATE workload, the data blocks that have been read into buffer cache are accelerated by EMC XtremCache. Meanwhile, the dirty blocks that have been flushed by DBWR are passing through EMC XtremCache and are directly written to the back-end VMAX array.

Uses the following EMC technology enablers in the reference architecture:

EMC VMAX 40K

XtremSF PCIe flash card

XtremCache caching software

This solution provides a foundation that can be scaled in a flexible, predictable, and nearly linear way using additional server resources, including CPUs and memory, HBA ports, and front-end ports, to provide higher IOPS and throughput based on the configuration in this solution.



8

Introduction

The purpose of this white paper is to describe an EMC High Performance Solution for Oracle on VMAX infrastructure based on an open architecture, and to demonstrate the following capabilities of the infrastructure:

High performance and flexibility

Low operational costs

Reduced risk

This solution validates the performance of the solution and provides guidelines to build similar solutions. This document is not a comprehensive guide to every aspect of this solution.

This white paper serves the following purposes:

Introduces the key solution technologies

Describes the solution architecture and design

Describes the solution test scenarios and present the results of performance testing

Identifies the key business benefits of the solution

This white paper is intended for chief information officers (CIOs), data center directors, Oracle DBAs, storage administrators, system administrators, technical managers, and any others involved in evaluating, acquiring, managing, operating, or designing Oracle database environments.

Purpose

Scope

Audience



Table 2 lists the terminology used in this solution paper.

Table 2. Terminology

Term Definition

AWR Automatic Workload Repository

ASM Automatic Storage Management

DML Data Manipulation Language

IOPS I/Os Per Second

LUN Logical Unit Number

OLTP Online Transaction Processing

PCIe Peripheral Component Interconnect Express

PGA Program Global Area

RAID Redundant Array of Independent Disks

RAC Real Application Clusters

SATA Serial Advanced Technology Attachment

SLOB Silly Little Oracle Benchmark

SGA System Global Area

Terminology



10

Technology overview This section describes the architecture of the solution.

The solution provides an optimal cost-to-performance ratio for Oracle mission-critical application environments. The architecture is composed of the following layers:

Server–Eight Cisco UCS C240 M3 servers each with:

Two 700 GB EMC XtremSF SLC flash card (1400 GB total)

EMC XtremSF driver and firmware

Network–Two Cisco Director - MDS 9506 switches

Storage–EMC Symmetrix VMAX 40K storage system

Database–One eight-node Oracle RAC database is installed on the eight servers

Figure 1 depicts the EMC Proven High Performance Solution for Oracle RAC on VMAX.

Figure 1. Solution architecture

Architecture layers

Architecture diagram



EMC Oracle Performance Solution: Server layer Comprising the server layer of the solution, eight Cisco UCS C240 M3 servers utilize a total of 128 cores with 2.90 GHz E5-2690 processors, 2.56 TB RAM, and 11 TB XtremSF flash PCIe cards. The Cisco UCS C240 M3 is an enterprise-class rack server designed for performance and expandability. As part of the EMC Proven High Performance Solution for Oracle RAC on VMAX, the server layer enables a high-performing, consolidated approach to an Oracle infrastructure, resulting in deployment flexibility without the need for application modification. Features and benefits include the following:

Intel Xeon processor E5-2600 product family for hardened protection for virtual and cloud environments

Fully integrated quad-port gigabit Ethernet

Figure 2 shows one of the eight Cisco UCS C240 M3 rack servers used in the EMC Proven High Performance Solution for Oracle RAC on VMAX solution.

Figure 2. Cisco UCS C240 M3 rack server (1 of 8)



12

Table 3 describes the various hardware components of the EMC Proven High Performance Solution for Oracle on the VMAX server layer.

Table 3. Server hardware

Server

hardware Quantity Configuration Description

Cisco UCS C240 M3

8 2 x 8-Core Sandy-Bridge E5-2690 Processors – 512 GB RAM - 4x 200 GB SSD server

Servers

Each server consists of the following components:

PCIEHHS-7XXM

2 700 GB SLC PCIe card EMC XtremSF

UCSC-C240-M3S

1 UCS C240 M3 SFF w/o CPU mem HD PCIe with rail kit expdr

Server housing

UCS-CPU-E5-2690

16 2.90 GHz E5-2690/130W 4C/10 MB Cache/DDR3 1600 MHz

CPUs

UCS-ML-1X324RY-A

16 32 GB DDR3-1600 MHz LR DIMM/PC3-12800/quad rank/x4/1.35v

DRAM

UCS-SD200G0KA2-E

4 200 GB std height 15 mm SATA SSD hot plug/drive sled mounted

Internal hard drives

UCSC-SD-16G-C240

1 16 GB SD card module for C240 servers

SD card

UCSC-RAIL-2U

1 2U rail kit for UCS C-Series servers

Rail kit

N20-BBLKD 20 UCS 2.5-inch HDD blanking panel

HDD panels

UCSC-HS-C240M3

2 Heat sink for UCS C240 M3 rack server

Heat sinks

UCSC-PCIF-01F

4 Full-height PCIe filler for C-Series

PCI slot fillers

UCSC-PCIF-01H

1 Half-height PCIe filler for UCS PCI slot filler

UCSC-RAID-11-C240

1 LSI 2008 SAS RAID Mezzanine Card for UCS C240 server

RAID card

CAB-C13-C14-AC

2 Power cord C13 to C14 (recessed receptacle) 10A

Power cables

UCSC-PSU-650W

2 650 W power supply for C-Series rack servers

Power supplies

Server hardware



Server


LPE12004-M8

2 Emulex Quad Channel 8Gb FC PCIe HBA

Fibre Channel cards

E10G42BTDA 1 Intel X520-DA2- Network adapter - PCI Express 2.0 x8 low profile - 10 gigabit Ethernet - 2 ports

10 GbE network card

E10GSFPSR 2 Intel Ethernet SFP+ SR Optics - SFP+ transceiver module - 1000Base-SX, 10GBase-SR - 850 nm

Optical ports for FC

Table 4 describes the various software components of the solution server layer.

Table 4. Server software

Server software Configuration Description

Red Hat Enterprise Linux 6.3 Operating system for database servers

Oracle Grid Infrastructure 11g Release 2

Enterprise Edition 11.2.0.3

Software provides Clusterware and ASM storage volume management

Oracle Database 11g Release 2

Enterprise Edition 11.2.0.3

Oracle Database software

EMC XtremCache software

2.0.1 EMC XtremCache software for server-side flash cache

Red Hat Enterprise Linux includes enhancements and new capabilities, including developer tools, virtualization features, security, scalability, file systems, and storage. Red Hat Enterprise Linux is a versatile platform that can be deployed on physical systems, as a guest on the major hypervisors, or in the cloud. It supports all leading hardware architectures with compatibility across releases.

Server software

Red Hat Enterprise Linux



14

Oracle Database 11gR2 is available in a variety of editions tailored to meet the business and IT needs of an organization. This solution utilizes Oracle Database 11gR2 Enterprise Edition (EE). Oracle Database 11g R2 EE delivers industry-leading performance, scalability, security, and reliability on a choice of clustered or single servers running Windows, Linux, or UNIX. The database supports advanced features such as Virtual Private Database, and data warehousing options such as partitioning and Advanced Analytics, that are either included or available as extra-cost options.

XtremSF and XtremCache overview In this solution, two 700 GB EMC XtremSF flash cards are used in each RAC node, and one XtremCache cache device is created from one XtremSF card.

EMC XtremSF is an advanced flash storage technology deployed in a server and designed to deliver unprecedented performance acceleration by reducing latency and increasing I/O throughput. EMC XtremSF allows applications to access data in the most efficient manner possible. Residing on the server PCIe interconnect bus, XtremSF reduces application response time from milliseconds to microseconds by caching the frequently accessed data at the server side, which reduces the number of I/O requests sent to backend disk array.

EMC XtremCache is server flash-caching software that reduces latency and accelerates throughput to dramatically improve application performance when coupled with EMC XtremSF PCIe flash technology. XtremCache accelerates reads and protects data by using a write-through cache policy to the networked storage to deliver persistent high availability, integrity, and disaster recovery.

XtremCache coupled with array-based EMC FAST™ software provides the most efficient and intelligent I/O path from the application to the underlying disk array. The result is a networked infrastructure that is dynamically optimized for performance, intelligence, and protection for both physical and virtual environments.

Benefits of XtremCache include the following:

Provides performance acceleration for read-intensive workloads with the protection of a back-end, networked storage array

Provides an intelligent path for the I/O and ensures that the right data is in the XtremCache of the servers at the right time

By offloading flash and wear-level management onto the XtremSF PCIe flash card, uses minimal CPU and memory resources from the server

Works in both physical and virtual environments

Provides better data protection: since XtremCache is a write-through cache, it does not comprise data consistency in the storage array, even if the cards fail in the middle of I/O processing

No need to warm up the XtremCache across database instances reboot

Works for any kind of I/O; for example, any applications and any database platforms

Is supported on various operating systems and server platforms

Oracle Grid Infrastructure and Database 11g Release 2

XtremSF and XtremCache deployment



Allows customers flexibility in choice of cache capacity on the cards

Supports the Oracle RAC database, and even these RAC databases are “stretched” with EMC VPLEX

As XtremCache is installed in a greater number of servers in the environment, more I/O processing is offloaded from the storage array to XtremCache configured on the servers. This provides a highly scalable performance model in the storage environment. For more detail, refer to:

Introduction to EMC XtremCache for Oracle Real Application Clusters listed in References

Introduction to EMC XtremCache for Oracle Real Application Clusters video listed in References

EMC XtremCache configuration

XtremCache supports Oracle RAC using a distributed cache coherency algorithm. XtremCache automatically recognizes the presence of Oracle RAC, and switches operation to clustering mode.

All working Oracle RAC nodes must have XtremCache installed for the distributed Cache feature to come online. EMC recommends using XtremCache with Oracle RAC to cache LUNs that holding data files and TEMP files. Do not use XtremCache to cache redo logs, archives, or clusterware files.

Steps for configuring XtremCache devices are shown in the XtremCache manual.



16

EMC Oracle Performance Solution: Network layer The switch component level is made up of two Cisco MDS 9506 director-class SAN switches (like the one shown in Figure 3). The Cisco MDS 9506 is designed for deployment in storage networks supporting virtualized data centers and enterprise clouds. The switch combines high performance and low total cost of ownership, a core architectural requirement at all levels of the VMAX Performance Block.

Figure 3. Cisco MDS 9506 Multilayer Director

The Cisco MDS 9506 also offers these benefits:

Highly available scalability through a combination of nondisruptive software upgrades, stateful process failover, and full redundancy of all core components

Superior platform for accelerated, intelligent storage applications such as EMC replication and backup, data migration, and storage media encryption

Virtual machine transparency and end-to-end visibility all the way from the virtual machine down to the EMC storage, enabling scalable, mobile virtual machines

Nexus 5.2(8) software is used in the EMC High Performance Solution for Oracle on VMAX.

Table 5 lists the hardware components of the network layer of this solution.



Table 5. Hardware components of the network layer

Network


MDS-PBF-ADV32 8 Cisco 32-port 8-Gbps FC Port Module

Line cards

MDS-9506-V2 2 Chassis SUP2 no ports Director Director chassis

FC10M-50MLC 128 FCHNL 10M 50/125 LC-LC LC adapters

MDS-PW19-TWST 2 Cisco 9506 Twist Lock Power Cord US

Power cord

FC1M-50MLC 96 FCHNL 1M 50/125 LC-LC

MDS-8G-SW 192 MDS 2/4/8-Gbps FC shortwave SFP LC

MDS-ENT-9500 2 Enterprise license key 9500



18

EMC Oracle Performance Solution: Storage layer

EMC is not only the world’s largest storage provider, but also the leader in storage infrastructure in Oracle Database environments. Storage User Demand Study, 2012 – Fall Edition: Many Uses of Internal Storage provides more details.

The storage components comprising the EMC Proven High Performance Solution for Oracle on VMAX include the following:

EMC Symmetrix VMAX 40K with eight engines (the specification of the engine is shown in Figure 4)

EMC PowerPath®

Dynamic cache partitioning (VMAX)

Figure 4. EMC Symmetrix VMAX 40K

Table 6. EMC Symmetrix VMAX storage hardware

Storage


SD-DBV-DIR-1P 4 VMAX 40K DBV DIR 1 Phase

SD-DBV-SPS 16 VMAX 40K SPSV

SD-1P 1 VMAX 40K 1P INFRAST

SVDBSOLDOR1P 4 VMAX 40K Drive Bay Solid Door 1P

SYMV2-MIGRBAS 1 Symmetrix 40K Migration Bundle

PP-SE-SYM 1 PPATH SE SYM

Introduction

Storage components

Storage hardware

http://www.emc.com/collateral/analyst-reports/suds-fall-edition-june-2012.pdf

http://www.emc.com/collateral/analyst-reports/suds-fall-edition-june-2012.pdf



Storage


SD-VCONFIG32 1 VMAX 40K VCONFIG 32

SD-FE80000E 16 VMAX 40K 8 MM 8 G Fibre Fibre Ports

SD-INTBKVKIT 8 VMAX 40K Internal Cable Bracket Titan

SD-PW40U-US 10 30A 1Phase Namer Japan l6-30P Power

VL4103001B 376 VMAX 40K 4 G 10K 300 GB SAS Drive

Disk drives

SD-ADD192C 7 VMAX 40K Add Engine-192GB-C VMAX engine

SD-192-BASEC 1 VMAX 40K Base-192GB-C VMAX cache

SD-DE25-DIR 64 VMAX 40K 25SLT DR ENCL

The EMC Symmetrix VMAX 40K is designed for high efficiency, scalability, and secure data persistence. Built on the strategy of powerful, trusted, and smart storage, and founded in EMC Virtual Matrix Architecture that allows for seamless, cost-effective growth, the EMC Symmetrix VMAX 40K offers the following:

Zero downtime migration technology and lower cost and greater efficiency through automated tiering

More scalability for less management complexity and operational expense

The configuration supporting this solution is described in Table 7:

Table 7. EMC Symmetrix VMAX 40K configurations

Component Quantity Configuration

Engines 8 192 GB cache each, total of 1536 GB raw cache

Directors 16 Eight ports on each director with 8 GB Fibre Channel

Bays 5 1 system, 4 disk

10K SAS drives 376 100 TB Raw, 45 TB Usable (RAID1 Configured)

EMC Symmetrix VMAX 40K eight-engine configuration



20

Table 8 lists the software used in the solution storage layer.

Table 8. Storage software

Storage software Configuration Description

VMAX Enginuity™ code 5876 VMAX micro code

EMC Solutions Enabler 7.6 Host CLI storage management software

EMC PowerPath 5.7 SP1 Multipathing and load balancing software

Storage software



EMC Oracle Performance Solution: Oracle Database layer

In Oracle 11g R2, Oracle ASM and Oracle Clusterware have been integrated into the Oracle Grid Infrastructure. In the solution, we use ASM to store the database files.

EMC Virtual Provisioning™ automatically stripes data across all data devices in a virtual pool and balances the workload across storage devices. To ensure even striping of data, all data devices in a virtual pool should be the same size.

Table 9 shows the RAID selections and number of spindles for each virtual pool. In this solution, Oracle data files and redo log files are located on thin devices using RAID 1 protection and all physical spindles for the best performance and capacity.

Table 9. Virtual pool design on VMAX 40K

Virtual pool

RAID protection

Drive type

Physical spindles size

Number of active spindles Item

FC_R1_1 RAID 1 (2-way, mirror)

FC 10K 300 GB 376+ 5 (spare disk)

CRS, DATA, REDO

Table 10 details the RAC database Oracle ASM (automatic storage management) disk group design. On each database, we used three ASM disk groups to store the relevant database files, including data files, control files, online redo log files, and temporary files. Default settings are used for ASM disk groups.

Table 10. ASM disk group design for RAC database

Item LUN size (GB) Number of LUNs ASM disk group name

CRS 10 2 +CRS

DATA 1024 18 +DATA

REDO 60 4 +REDO

Introduction

Storage virtual provisioning design

ASM disk group configuration for OLTP database



22

Table 11 details each OLTP database workload profile for the solution. We used the SLOB toolkit to generate an OLTP database and deliver the OLTP-like workloads, including the query-only and UPDATE-only workloads required for the solution.

Table 11. Database workload profile for each OLTP database

Profile characteristic Details

Database type OLTP

Database size 16 TB

Oracle Database 11gR2 eight-node RAC database on ASM

Instance configuration for read only workload

SGA size: 16 GB

Note: Because more SGA causes less I/O activity, we reduced SGA to this small number to generate a stable and high I/O workload

Workload profile OLTP-like workload simulated by SLOB

Network connectivity 8 Gb Fibre Channel for SAN; 10 GbE for IP

OLTP database and workload profile



EMC Oracle Performance Solution: Performance testing

The solution characterizes the Oracle OLTP system performance on VMAX array with EMC XtremSF cards installed on database servers. We created an eight-node Oracle RAC database on eight physical servers. Silly Little Oracle Benchmark (SLOB) was used to generate workload because it is the premiere SQL workload for driving maximum physical random I/O from a database platform.

SLOB is a SQL-driven Oracle database I/O generator, instead of a synthetic I/O generator. SLOB uniquely drives massive physical I/O using minimal host CPU resources, and it specifically targets the Oracle I/O subsystem. SLOB performs all of its physical I/O buffered in the Oracle SGA. No physical I/O buffered in the Oracle PGA is performed. SLOB possesses the following characteristics:

Supports testing Oracle logical read (SGA buffer gets) scaling

Supports testing physical, random single-block reads (db file sequential read/db file parallel read)

Supports testing random single block writes (db file parallel write)

Supports testing extreme REDO logging I/O

Consists of simple PL/SQL

Is entirely free of all application contention

We used SLOB to generate an OLTP-like workload on the eight-node Oracle RAC database to demonstrate sustained flash and storage array IOPS. The database performance metrics including IOPS and latency were gathered primarily from the AWR report. In addition, we gathered metrics for I/O throughput at the server/database and storage levels.

Notes: Benchmark results are highly dependent on workload, specific application requirements, and system design and implementation. Relative system performance will vary because of these and other factors. Do not use solution test workloads as a substitute for a specific customer application benchmark when making critical capacity planning or product evaluation decisions All performance data contained in this report was obtained in a rigorously controlled environment. Results obtained in other operating environments may vary significantly. EMC Corporation does not warrant or represent that a user can or will achieve similar performance expressed in transactions per minute.

Overview



24

This test measures the physical I/O scalability along with the scaling of the number of concurrent SLOB zero-think-time sessions (simulated concurrent users) and the number of RAC nodes. Multiple concurrent sessions (reader sessions) executing similar query SQL statements were run to validate read only workload, and multiple concurrent sessions (writer sessions) executing similar UPDATE SQL statements were run to validate the physical read/write workload.

We gradually increased the number of instances of the RAC database and the number of concurrent users with each user running similar OLTP queries simultaneously.

When we added a RAC node, additional resources were added, including CPU power and XtremSF cards. With the addition of new RAC nodes, we tested the system again by running the similar SLOB workload. For this test, workloads were running simultaneously from all the RAC nodes added. We then increased the number of concurrent users and measured the performance scalability.

The test process included the following steps:

1. Run the OLTP query only workload with 64 concurrent simulated users (zero-think-time sessions) on the first node of an eight-node RAC database using SLOB.

2. Add the second node into the system, then run the workload with 64 concurrent users on each node; that is, with 128 concurrent users in total running simultaneously on the two-node RAC database.

3. Add two additional nodes into the system, then run the workload with 64 concurrent users on each node separately; that is, with a total of 256 concurrent users running simultaneously on the four-node RAC database.

4. Add four additional nodes into the system, then run the workload with 64 concurrent users on each node separately; that is, with a total of 512 concurrent users running simultaneously on the eight node RAC database.

Performance statistics were captured using Oracle Automatic Workload Repository (AWR) RAC reports. We observed the “physical reads” value in the AWR report to assess read IOPS statistics. Query average response time was calculated from the “db file parallel read” and “db file sequential read” record in the “Top Timed Events” section of the AWR report, as shown in Figure 5.

Figure 5. AWR RAC report snippet for read IO response time calculation

Test objective

Query only test scenarios and methodology

Read test results



We used the following formula to calculate the I/O latency:

For “db file sequential read”

The total wait time is T1 which is 19,485.48 seconds as shown in Figure 5.

The total number of waits is N1 which is 30,614,070 as shown in Figure 5.

For “db file parallel read”

The total wait time is T2 which is 9,803.47 seconds as shown in Figure 5.

The total number of waits is N2 which is 8,610,615 as shown in Figure 5.

The average read response time is: (T1+T2) / (N1+N2), which is (19,485.48 + 9,803.47) * 1,000/ (30,614,070 + 8,610,615) as shown in Figure 5, and 0.75ms is the average response time.

Table 12 depicts how IOPS increase when RAC nodes scale up.

Table 12. Scaling of nodes and resulting increases in IOPS

Figure 6 shows that IOPS increase when the number of RAC nodes scales, while average read I/O response time remains under one millisecond.

Figure 6. Query only IOPS scaling and average physical read I/O response time

As Figure 6 shows, we achieved a total of 3,765,176 read IOPS and an average latency of 0.75 milliseconds with eight-node RAC database when running 64 concurrent sessions executing similar queries SQL statements on each of the RAC nodes.

Metrics 1 node 2 nodes 4 nodes 8 nodes

IOPS 457,136 962,155 1,914,963 3,765,176

Average response time (ms) 0.74 0.69 0.68 0.75



26

The IOPS increased nearly linearly with each additional node that was added into the Oracle RAC database. For example, the total IOPS of four RAC nodes reached 1,914,963. After another four RAC nodes are added for a total of eight database nodes, the IOPS are almost doubled to 3,765,176.

The read hit ratio for XtremCache is about 98 percent for each cache device during the test. Two percent of the I/Os are served from the storage array. The statistics can be monitored by running the following command line:

vfcmt display -cache_dev <device>

We gradually increased the number of RAC nodes and ran a total of 56 concurrent sessions on the database to generate stable and high read and write IOPS, with each session running similar SLOB UPDATE SQL statements

We decreased buffer cache for each database instance to push consistent write I/O workload to the back-end storage. The write workload was driven by the UPDATE SQL statement. Generally, it incurs the following operations sequentially:

1. Read the data blocks that need to be updated into the buffer cache.

2. Update the rows in the data blocks.

3. Commit the updated rows and trigger LGWR flushing redo entries to online log files.

During the SQL UPDATE only workload running, the background DBWR process flushes the dirty blocks out of the buffer cache into data files. Because we used a very small buffer cache, the data blocks were read into the buffer cache and written out of the buffer cache soon after the rows were updated. Thus the execution of each UPDATE operation caused physical reads, which were accelerated by the EMC XtremCache (when cache hit) or the back-end VMAX array (cache miss). When the updated data blocks were written out of the buffer cache by the DBWR process, as being a write through cache for EMC XtremCache , the data blocks were written to the back-end VMAX array. Once the application got the acknowledgement from the back-end array, the application I/O request was complete, and the updated data was asynchronously written to XtremCache.

The test process included the following steps:

1. Run the Update only workload with 56 concurrent users on one RAC Database with only one node using the scripts in SLOB.

2. Add one additional node into the RAC database, then run the workload with 56 concurrent users; that is, with 28 concurrent users running simultaneously on each node of this two-node RAC.

3. Add two additional nodes into the RAC database, then run the workload with 56 concurrent users; that is, with 14 concurrent users running simultaneously on each node of this four-node RAC.

4. Add four additional nodes into the RAC database, then run the workload with 56 concurrent users; that is, with seven concurrent users running simultaneously on each node of this eight-node RAC

Update only test scenarios and methodology



Performance statistics are captured using AWR reports. We read the “physical writes” row in the AWR report for the peak write IOPS statistics. Because the write workload is generated by UPDATE statements as described previously, we also collected “physical reads” from the AWR report for the peak read IOPS statistics that were part of the write transaction.

We calculated updated write average response time by dividing the “Total Wait Time(s)” by the “Waits” of the “db file parallel write” record in the “Top Timed Events” section of the AWR report, as shown in Figure 7. We also calculated LGWR latency by dividing the “Total Wait Time(s)” by the “Waits” of the “log file parallel write” record in the “Top Timed Events” section of the AWR report, as shown in Figure 7.

For example, in following AWR snippet, the total wait time of the “db file parallel write” wait event is 287.00 seconds, which is 287,000 ms, and the number of waits is 319,941; thus, the write average response time can be calculated as 287,000 / 319,941= 0.90 ms. The LGWR latency is: 1,087.81*1,000/1,323,505 = 0.82.

Figure 7. Update only write average response time measurement from the RAC AWR report

Table 13 and Figure 8 depict the peak read/write IOPS increases as the RAC node scaling.

Table 13. Scaling of users and resulting increases in peak disk array IOPS

IOPS 1 node 2 nodes 4 nodes 8 nodes

Write 26,950 50,110 95,932 153,276

Read 26,542 49,539 94,877 150,054

Aggregate 53,492 99,649 190,809 303,330

Write response time

0.30 0.40 0.30 0.90

Read response time(ms)

0.90 0.80 0.20 0.30

Redo size (MB/s) 20 37 71 115

LGWR latency (ms)

0.55 0.54 0.66 0.82

Peak write test results



28

Figure 8 shows the peak IOPS, and update-only IOPS for RAC node scaling, while average response time remains under one millisecond.

Figure 8. Peak writes – update-only IOPS scaling along with RAC nodes scaling

During an UPDATE transaction, the backend VMX only needs to handle the writing I/O activities, as the reading I/O activities have been cached and accelerated by XtremCache. Because of this, the solution can scale to accommodate a very heavy transaction workload, as confirmed in testing.

As shown in Table 13 and Figure 8, when running 56 concurrent sessions executing similar UPDATE SQL statements on eight-node RAC database, we achieved 303,330 aggregated IOPS including 153,276 write IOPS and 150,054 read IOPS, which were part of the write transaction. The average latency of write was 0.9 milliseconds, because we used very small buffer cache (only 128 MB, almost no data cached in the server) to generate a high physical write I/O workload.

The IOPS increased nearly linearly when additional RAC nodes were added into the workload. For example, the aggregate IOPS were 53,492 when running write workload on one RAC node, and this increased to 99,649 when running workload on two RAC nodes.

Redo size is also a key metric used to measure the transaction capability. As demonstrated through testing, the workload on one node generated 20 MB/second redo entries, and almost doubled to 37 MB/second with the workload running on two nodes. When we ran workload on four nodes the redo throughput is almost doubled again to 71 MB/second. The transaction capability is scaling along with the node scaling.



Conclusion

Implementing the EMC Proven High Performance Solution for Oracle RAC on VMAX with innovative and proven products like EMC’s VMAX gives customers choices within an open infrastructure, enabling them to easily integrate into existing data center operations and take advantage of new technologies such as XtremCache, which now also fully supports Oracle RAC. Customers can efficiently use people, processes, and technologies through virtualization, database, and applications consolidation. Customers can also independently scale capacity and processing capability without the limitations imposed by a single-purpose appliance

As the infrastructure undergoes changes to applications, databases, and non-database software, this stack can align with the shifting technical demands imposed by the business needs. This white paper explains how to balance OLTP workloads while maintaining the protection and resiliency of the data. That adaptability to change and the ability to apply the technology where it is needed protects the capital investment, and can be fluid as the requirements change without sacrificing any of the other datacenter operations.

The core advantages of this solution are as follows:

With its open architecture, supports Oracle OLTP workload built on superior functional integration throughout the solution

Leverages the following EMC technology enablers in the solution:

EMC VMAX 40K

XtremSF flash cards

XtremCache

Provides full support for EMC Performance Boost, HA, and replication technologies

Increases IOPS for OLTP workloads when servers are added

The aggregate read IOPS increased from 457,136 to 1,914,963 when RAC nodes were scaled out from one to four, and it increased to 3,765,176 when eight RAC nodes ran the workload together.

Average response time less than one millisecond (0.75ms)

The aggregate IOPS for UPDATE transaction workload was 53,492 on one RAC node and caused 20 MB/s redo throughput, and it increased to 190,809 and caused 71 MB/s redo throughput when four RAC nodes were added. The IOPS increased to 303,330 and generated 115 MB/s redo throughput when running on eight RAC nodes.

Average response time for DBWR background process less than one millisecond (0.90 ms)

This solution provides a foundation that can be scaled in a flexible, predictable, and near-linear way, by adding additional server resources including CPUs and memory, HBA ports, and front-end ports, to provide higher IOPS and throughput based on the described configuration.

Summary

Findings



30

References

The following documents provide additional and relevant information. Access to these documents depends on your login credentials. If you do not have access to a document, contact your EMC representative.

The following reference materials are available on EMC Powerlink:

EMC Infrastructure for High Performance Microsoft and Oracle Database Systems

Introduction to EMC XtremCache

EMC XtremCache Data Sheet

In addition, XtremCache documentation is available at EMC Online Support: https://support.emc.com/products/25208_XtremCache-Cache/Documentation/

A video entitled Introduction to EMC XtremCache for Oracle Real Application Clusters is visible via this link: https://community.emc.com/videos/6740

For additional information, see the following documents:

Oracle Grid Infrastructure Installation Guide 11g Release 2(11.2) for Linux

Oracle Database Installation Guide 11g Release 2 (11.2) for Linux.

EMC documentation

Oracle documentation

https://support.emc.com/products/25208_XtremSW-Cache/Documentation/

https://community.emc.com/videos/6740

high performance solution for oracle 11g rac with ... · high performance solution for oracle 11g...

Documents