using vplex metro and recoverpoint with vmware srm in an oracle extended rac e ... ·...

White Paper

Abstract

This white paper discusses how to configure a VPLEX Metro with Oracle Applications on Oracle Extended RAC to replicate to a third site with RecoverPoint and VMware vCenter Site Recovery Manager for HA and DR. July 2013

USING VPLEX METRO AND RECOVERPOINT WITH VMWARE SRM IN AN ORACLE EXTENDED RAC E-BUSINESS SUITE ENVIRONMENT DEMONSTRATING INTEROPERABILITY OF VPLEX, RECOVERPOINT AND VMWARE SRM WITH ORACLE APPLICATIONS AND EXTENDED RAC FOR HA AND DR

2 Using VPLEX Metro and RecoverPoint with VMware SRM in an Oracle Extended RAC E-Business Suite Environment

Copyright © 2013 EMC Corporation. All Rights Reserved. EMC believes the information in this publication is accurate of its publication date. The information is subject to change without notice. The information in this publication is provided “as is”. EMC Corporation makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any EMC software described in this publication requires an applicable software license. For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com. VMware, ESXi, vMotion, and vSphere are registered trademarks or trademarks of VMware, Inc. in the United States and/or other jurisdictions. All other trademarks used herein are the property of their respective owners. Part Number h11767.1


Table of Contents

Executive summary.................................................................................................. 6

Audience ............................................................................................................................ 6

Terminology ....................................................................................................................... 6

Document scope and limitations ........................................................................................ 7

Introduction ............................................................................................................ 8

Symmetrix VMAX with Enginuity 5876 ...................................................................... 8

EMC Unisphere for VMAX and VPLEX ........................................................................ 8

EMC Virtual Storage Integrator ............................................................................... 11

VMware ESXi version 5.x with PowerPath/VE .......................................................... 14

Symmetrix Virtual Provisioning .............................................................................. 15

Fully Automated Storage Tiering (FAST) .................................................................. 17

FAST and Fully Automated Storage Tiering with Virtual Pools (FAST VP) ............................. 17

FAST managed objects ..................................................................................................... 17

FAST VP allocation by FAST Policy .......................................................................... 18

Oracle Applications ............................................................................................... 18

VPLEX ................................................................................................................... 19

Mobility ........................................................................................................................ 20

Availability ................................................................................................................... 20

VPLEX Product Offerings ................................................................................................... 20

Logging volumes .............................................................................................................. 22

Redundancy with RecoverPoint ......................................................................................... 22

VASA with VPLEX .............................................................................................................. 23

FAST VP and Oracle Applications 12 ....................................................................... 24

Applications Architecture ................................................................................................. 24

Working with FAST VP and Oracle Applications on VMware vSphere ................................. 25

Oracle Applications Tablespace Model ............................................................................. 26

Oracle Applications implementation ................................................................................. 26

Oracle Applications deployment ....................................................................................... 27

Hardware layout ............................................................................................................... 28

FAST VP in the VPLEX Metro Cluster ........................................................................ 28

RecoverPoint Remote Site ................................................................................................. 33

Device Pinning in FAST VP for the Oracle Database ................................................. 34

Pinning the REDO devices in FAST VP ................................................................................ 35


VPLEX Distributed Devices and Consistency Groups ............................................... 36

Provisioning Storage example .......................................................................................... 37

VPLEX Virtual Volume Wizard ........................................................................................ 38

VPLEX Distributed Device Creation ............................................................................... 45

Failure status of distributed device ............................................................................... 49

Enabling RecoverPoint on the VPLEX consistency group ............................................... 50

Adding distributed devices to storage views ................................................................. 51

Viewing in VMware ....................................................................................................... 54

VPLEX Local Devices for RecoverPoint Journals ....................................................... 56

Consistency Group creation for RecoverPoint Journal .................................................... 57

Adding RecoverPoint Journal to a storage view ............................................................. 59

VMware Implementation ........................................................................................ 60

VMware deployments in a VPLEX Metro environment ........................................................ 60

VPLEX Metro Witness ........................................................................................................ 60

VPLEX Failure Handling ................................................................................................. 63

VMware Cluster Configuration with VPLEX Metro Witness ........................................ 63

VMware cluster configuration ....................................................................................... 63

VMware DRS Groups and Rules .................................................................................... 64

Best practices with VMware HA and VPLEX Metro ............................................................. 69

Oracle Real Application Clusters (RAC) on VPLEX Metro ........................................... 69

Extended RAC and Oracle Clusterware with VPLEX Witness ............................................... 69

Oracle on VMware ............................................................................................................ 70

Oracle Real Application Clusters (RAC) and Oracle E-Business Suite ................................. 70

Oracle Real Application Clusters (RAC) on VMFS ............................................................... 70

Oracle Clusterware deployment .................................................................................... 71

Oracle Extended RAC installation ...................................................................................... 71

Procedure ..................................................................................................................... 71

Networking ....................................................................................................................... 73

EMC RecoverPoint ................................................................................................. 75

RPA .............................................................................................................................. 76

Volumes ....................................................................................................................... 76

Snapshot/PIT ............................................................................................................... 76

Logged (physical) access .............................................................................................. 77

Direct access ................................................................................................................ 77

RecoverPoint 4.0 .............................................................................................................. 77

RecoverPoint CDP and SRDF ......................................................................................... 77

VPLEX Metro RecoverPoint Configuration ................................................................ 77

VPLEX Metro and RecoverPoint CRR/CLR ........................................................................... 77

Unisphere for RecoverPoint .................................................................................... 79


RecoverPoint Configuration for the Oracle Environment .......................................... 81

Licensing RecoverPoint ..................................................................................................... 81

RecoverPoint Configuration.................................................................................... 85

RecoverPoint Journals ....................................................................................................... 85

RecoverPoint Replicas ...................................................................................................... 86

Consistency Group Creation ............................................................................................. 86

Setting external management for a consistency group .................................................. 91

Group Set creation ....................................................................................................... 92

VMware SRM Installation ....................................................................................... 94

RecoverPoint SRA ............................................................................................................. 94

VMware SRM Protection and Recovery Groups ........................................................ 95

Add a RecoverPoint Array Manager ................................................................................... 96

VMware SRM Protection Group creation ............................................................................ 98

VMware SRM Test Failover ................................................................................... 102

VSI: RecoverPoint Management .......................................................................... 102

RecoverPoint Failover Testing with VSI ................................................................. 104

Point-in-Time assignment in VSI for RecoverPoint Consistency Groups ........................... 104

Test Failover ............................................................................................................... 108

RecoverPoint PiT with VSI in Disaster Recovery ..................................................... 113

Enabling PiT with VSI RM on the Recovery Site ................................................................ 114

Recommendations when using RecoverPoint with VMware SRM ............................ 121

Conclusion .......................................................................................................... 121

References .......................................................................................................... 122

EMC ................................................................................................................................ 122

VMware .......................................................................................................................... 122

Oracle ............................................................................................................................ 123


Executive summary The EMC® VPLEX™ family removes physical barriers within, across, and between datacenters. VPLEX Metro provides data access and mobility between two VPLEX clusters within synchronous distances. With a unique scale-out architecture, VPLEX’s advanced data caching and distributed cache coherency provide workload resiliency, automatic sharing, balancing and failover of storage domains, and enable both local and remote data access with predictable service levels.

VMware vSphere™ makes it simpler and less expensive to provide higher levels of availability for important applications. With vSphere, organizations can easily increase the baseline level of availability provided for all applications, as well as provide higher levels of availability more easily and cost-effectively. vSphere makes it possible to reduce both planned and unplanned downtime. The revolutionary VMware vMotion™ (vMotion) capabilities in vSphere make it possible to perform planned maintenance with zero application downtime while VMware vCenter Site Recovery Manager enables those virtualized environments with a robust disaster recovery solution.

EMC RecoverPoint is an advanced enterprise-class disaster recovery solution supporting heterogeneous storage and server environments. RecoverPoint helps customers accelerate protection and provides operational and disaster recovery of their VMware vSphere environment, without impacting production environments. RecoverPoint is ideally suited for replicating and protecting virtualized environments.

Combining all these technologies together with an enterprise ERP and CRM software solution like Oracle Applications running on Oracle Extended RAC, provides a highly available, disaster recovery-enabled hardware and software solution that cannot be matched.

Audience

This white paper is intended for VMware administrators, Oracle DBAs, server administrators, and storage administrators responsible for creating, managing, and using VMware, as well as their underlying storage devices, for their VMware vSphere environments attached to a Symmetrix VMAX™ storage array running Enginuity™ 5876. The white paper assumes the reader is familiar with Oracle databases and applications, VMware environments, EMC Symmetrix VMAX, EMC VPLEX, EMC RecoverPoint, and the related software.

Terminology Term Definition

Device LUN, logical volume

Volume LUN, logical volume

Unisphere Unisphere for VMAX, VPLEX or RecoverPoint depending on

context

SYMCLI/CLI Solutions Enabler's Command Line Interface

Metavolume A collection of Symmetrix devices that represent one device at the

host level


Storage volume LUN or unit of storage presented by the back-end arrays to

VPLEX

Metadata volume VPLEX system volume that contains metadata about the devices,

virtual volumes, and cluster configuration

Extent All or part of a storage volume in a VPLEX configuration

Device VPLEX protection scheme applied to an extent or group of extents

Virtual volume Unit of storage presented by the VPLEX front-end ports to hosts

Front-end port Director port connected to host initiators (acts as a target) in

reference to a VPLEX

Engine Consists of two directors and is the unit of scale for the VPLEX

solution

VPLEX cluster A collection of VPLEX engines in one rack.

VPLEX Metro

The cooperation of two VPLEX clusters, each serving their own

storage domain over Synchronous distance forming an

active/active distributed volume

Oracle Extended RAC

A deployment model for Oracle Real Application Clusters in

which the server nodes reside in physical separate locations. Such

a deployment has strict software and hardware requirements but

can be enabled with such advanced technologies as VPLEX

Metro.

Acronym/Abbreviation Definition

LUN Logical Unit Number

TDEV Symmetrix Thin Device

FAST VP Fully Automated Storage Tiering with Virtual Pools

SG Storage Group

RDM Raw Device Mapping

VMFS VMware Virtual Machine File System

Document scope and limitations

This document applies to EMC VPLEX Metro configured with VPLEX Witness, EMC RecoverPoint, VMware vSphere and vCenter Site Recovery Manager, and Oracle Applications and Oracle RAC. The details provided in this white paper are based on the following software and hardware:

VPLEX GeoSynchrony 5.1 or higher

VPLEX Metro only (Local and Geo are not supported with FT or HA in a stretched configuration)

VPLEX Clusters are within 5 milliseconds (ms) of each other to allow for VMware HA

VPLEX Witness is deployed to a third failure domain. The Witness functionality is required for “VPLEX Metro” to become a true active/active continuously available storage cluster.


ESXi and vSphere 5.1

VMware vCenter Site Recovery Manager 5.1

Any qualified pair of arrays listed on the EMC Simple Support Matrix (ESSM), in this case Symmetrix VMAX.

EMC RecoverPoint 4.0 P1 and EMC RecoverPoint SRA 2.1

EMC VSI feature for RecoverPoint Management 5.5

Oracle Applications Release 12

Oracle RAC 11g R2

EMC Enginuity 5876 2013 Q2 SR with FAST VP

Introduction The purpose of this paper is not to present performance results or scalability tests of the various products included, but rather to lay out how these different technologies can be integrated and will work together to provide a robust High Availability and Disaster Recovery solution.

This document, therefore, will present the best practices for the various components involved in the solution, though the lab environment used may not have had access to all necessary components. Even were this lab environment to adhere perfectly to the physical requirements, it may not replicate a particular customer’s environment. By including the best practices, however, a customer can use the information and the resources at hand to design the best possible environment for their needs.

The details around setting up the physical hardware will only be mentioned as necessary to the solution, e.g. thin pools on the Symmetrix VMAX. Customers are advised to work with EMC to ensure their physical hardware for VPLEX and RecoverPoint are properly setup and configured to take advantage of solutions such as the one presented in this white paper.

Symmetrix VMAX with Enginuity 5876 The latest release of Enginuity 5876, 2013 Q2 SR (5876.229.145) carries the extended and systematic feature development forward from previous Symmetrix generations. This means all of the reliability, availability, and serviceability features, all of the interoperability and host operating systems coverage, and all of the application software capabilities developed by EMC and its partners continue to perform productively and seamlessly even as underlying technology is refreshed.

EMC Unisphere for VMAX and VPLEX


Beginning with Enginuity 5876, Symmetrix Management Console has been replaced by EMC® Unisphere™ for VMAX™ which offers big-button navigation and streamlined operations to simplify and reduce the time required to manage a data center. Unisphere for VMAX simplifies storage management under a common framework, incorporating Symmetrix Performance Analyzer which previously required a separate interface.

In addition, with the Performance monitoring option, Unisphere for VMAX provides tools for performing analysis and historical trending of Symmetrix system performance data.

The GUI interface dashboard is presented in Figure 1.

Figure 1. Unisphere for VMAX – Dashboard

Unisphere for VMAX, shown in the preceding figure, can be run on a number of different kinds of open systems hosts, physical or virtual. Unisphere for VMAX is also available as a virtual appliance for ESX(i) version 4.0 (and later) in VMware vSphere. For more details please visit Support.EMC.com.

Unisphere for VPLEX™, while still a separate application, provides the same GUI interface look and feel as Unisphere for VMAX, along with the same types of capabilities. Unisphere for VPLEX is accessed by making a secure (https) connection to the management server. In the case of VPLEX Metro, either management server provides access to both clusters. The user can move from claiming storage all the


way through presenting a distributed device to host. The system status page is in Figure 2. Note that the RecoverPoint Clusters status will always appear next to the cluster that is accessed through the management server regardless of where the RecoverPoint appliance (RPA) is configured.

Figure 2. Unisphere for VPLEX – System status

The Unisphere for VPLEX GUI contains several key performance statistics for host performance and overall health. They can be found on the Performance Dashboard tab and can be added to the default performance charts that are displayed. Using the data provided, the VPLEX administrator can quickly determine the source of performance problems within an environment. The dashboard is seen in Figure 3.


Figure 3. Unisphere for VPLEX – Performance dashboard

EMC Virtual Storage Integrator EMC Virtual Storage Integrator (VSI) for vSphere Client version 5.5 provides multiple feature sets including: Storage Viewer (SV), Path Management, Unified Storage Management, RecoverPoint Management, and Symmetrix SRA Utilities. Storage Viewer functionality extends the VMware vSphere Client to facilitate the discovery and identification of EMC Symmetrix, VPLEX, CLARiiON®, Isilon®, VNX® and Celerra® storage devices that are allocated to VMware ESX/ESXi hosts and virtual machines. Unified Storage Management simplifies the provisioning of Symmetrix VMAX virtual pooled storage for data centers, ESX servers, clusters, and resource pools. Path Management allows the user to control how datastores are accessed, while RecoverPoint Management and the SRA Utilities provide a framework for working with VMware vCenter Site Recovery Manager (VMware SRM) environments. VSI management is accessed through the home screen and the EMC VSI icon seen in Figure 4.


Figure 4. EMC VSI icon in the vSphere Client

Some of the VSI features are shown installed in Figure 5.


Figure 5. VSI 5.5 features

VSI for vSphere Client presents the underlying storage details to the virtual datacenter administrator, merging the data of several different storage mapping tools into a few, seamless vSphere Client views. VSI enables you to resolve the underlying storage of Virtual Machine File System and SAN datastores and virtual disks, as well as raw device mappings (RDM). In addition, you are presented with lists of storage arrays and devices that are accessible to the ESX(i) hosts in the virtual datacenter.

One of these features, the Storage Viewer, is displayed in Figure 6 and is demonstrating how to obtain detailed information about a LUN.


Figure 6. VSI 5.5 Storage Viewer feature

VMware ESXi version 5.x with PowerPath/VE EMC PowerPath®/VE delivers PowerPath multipathing features to optimize VMware vSphere virtual environments. PowerPath/VE enables standardization of path management across heterogeneous physical and virtual environments. PowerPath/VE enables one to automate optimal server, storage, and path utilization in a dynamic virtual environment. With hyper-consolidation, a virtual environment may have hundreds or even thousands of independent virtual machines running, including virtual machines with varying levels of I/O intensity. I/O-intensive applications can disrupt I/O from other applications, and before the availability of PowerPath/VE, as discussed in previous sections, load balancing on an ESX host system had to be manually configured to correct for this. Manual load-balancing operations to ensure that all virtual machines receive their individual required response times are time-consuming and logistically difficult to effectively achieve.

PowerPath/VE works with VMware ESXi as a multipathing plug-in (MPP) that provides enhanced path management capabilities to ESX and ESXi hosts. Previous versions of ESX do not have the PSA, which is required by PowerPath/VE

PowerPath/VE installs as a kernel module on the vSphere host. As shown in Figure 7, PowerPath/VE plugs in to the vSphere I/O stack framework to bring the advanced multipathing capabilities of PowerPath – dynamic load balancing and automatic failover – to VMware vSphere.

Figure 7. PowerPath/VE vStorage API for multipathing plug-in


When using PP/VE on ESXi hosts attached to VPLEX, the default optimization mode for VPLEX devices is ADaptive. The default optimization mode is the most appropriate policy for most workloads and should not be changed. The policy is shown in Figure 8.

Figure 8. PP/VE default policy for VPLEX of Adaptive

Using PP/VE with VMware on VPLEX is a best practice.

Symmetrix Virtual Provisioning Symmetrix Virtual Provisioning™, a type of host-accessible device called a thin device, or virtual device, that can be used in many of the same ways that regular, host-accessible Symmetrix devices have traditionally been used. VP offers storage efficiency/flexibility that traditional thick volumes cannot. Unlike regular Symmetrix devices, thin devices do not need to have physical storage completely allocated at the time the devices are created and presented to a host. The physical storage that is used to supply drive space for a thin device comes from a shared thin pool that has been associated with the thin device. A thin pool is comprised of internal Symmetrix devices called data devices that are dedicated to the purpose of providing the actual physical storage used by thin devices. When they are first created, thin devices are not associated with any particular thin pool. An operation referred to as “binding” must be performed to associate a thin device with a thin pool.

Figure 9 depicts the relationships between thin devices and their associated thin pools. There are nine devices associated with thin Pool A and three thin devices associated with thin pool B.


Figure 9. Thin devices and thin pools containing data devices

When a write is performed to a portion of the thin device, the Symmetrix allocates a minimum allotment of physical storage from the pool and maps that storage to a region of the thin device including the area targeted by the write. The storage allocation operations are performed in small units of storage called a “thin extent.” A round-robin mechanism is used to balance the allocation of thin extents across all of the data devices in the pool that are enabled and that have remaining unused capacity. A thin extent size is comprised of twelve 64 KB tracks (768 KB). That means that the initial bind of a thin device to a pool causes one extent, or 12 tracks, to be allocated per thin device.

When a read is performed on a thin device, the data being read is retrieved from the appropriate data device in the storage pool to which the thin device is bound. Reads directed to an area of a thin device that has not been mapped do not trigger allocation operations. The result of reading an unmapped block is that a block in which each byte is equal to zero will be returned. When more storage is required to service existing or future thin devices, data devices can be added to existing thin pools. New thin devices can also be created and associated with existing thin pools.

A thin device is always bound to a single thin pool but may have extents allocated in multiple pools within a single Symmetrix. A thin device may also be moved to a different thin pool, without any loss of data or data access, by using Virtual LUN VP Mobility. Virtual LUN VP Mobility provides the ability to migrate a thin device from one thin pool to another. If the LUN to move is part of a FAST VP Policy, it may only be moved to one of the thin pools in the policy.


Fully Automated Storage Tiering (FAST) Fully Automated Storage Tiering (FAST) automates the identification of data volumes for the purposes of relocating application data across different performance/capacity tiers within an array, or to an external array using Federated Tiered Storage (FTS).1

The primary benefits of FAST include:

Elimination of manually tiering applications when performance objectives change over time

Automating the process of identifying data that can benefit from Enterprise Flash Drives or that can be kept on higher-capacity, less-expensive SATA drives without impacting performance

Improving application performance at the same cost, or providing the same application performance at lower cost. Cost is defined as acquisition (both hardware and software), space/energy, and management expense

Optimizing and prioritizing business applications, allowing customers to dynamically allocate resources within a single array

Delivering greater flexibility in meeting different price/performance ratios throughout the lifecycle of the information stored

FAST and Fully Automated Storage Tiering with Virtual Pools (FAST VP)

EMC Symmetrix FAST (FAST DP) and FAST VP automate the identification of data volumes for the purposes of relocating application data across different performance/capacity tiers within an array. FAST operates on standard Symmetrix devices. Data movements executed between tiers are performed at the full volume level. FAST VP operates on virtual devices. As such, data movement execution can be performed at the sub-LUN level, and a single thin device may have extents allocated across multiple thin pools within the array. Because FAST DP and FAST VP support different device types – standard and virtual respectively – they both can operate simultaneously within a single array.2 Aside from some shared configuration parameters, the management and operation of each are separate.

FAST managed objects

There are three main elements related to the use of both FAST and FAST VP on Symmetrix VMAX, graphically depicted in Figure 10. These are:

Storage tier — A shared resource with common technologies

1 Other than the brief overview provided, Federated Tiered Storage will not be addressed in this particular whitepaper. For more information on FTS, refer to the Design and Implementation Best Practices for EMC Symmetrix Federated Tiered Storage (FTS) technical note available at Support.EMC.com. 2 This holds true for all the Symmetrix family except the VMAXe/VMAX 10K which only supports FAST VP, being a completely thin-provisioned array.


FAST policy — Manages a set of tier usage rules that provide guidelines for data placement and movement across Symmetrix tiers to achieve service levels and for one or more storage groups

Storage group — A logical grouping of devices for common management

Figure 10. FAST managed objects

Each of the three managed objects can be created and managed by using either Unisphere for VMAX (Unisphere) or the Solutions Enabler Command Line Interface (SYMCLI).

Note: For more information on FAST VP specifically please see the technical note FAST VP for EMC Symmetrix VMAX Theory and Best Practices for Planning and Performance available at Support.EMC.com.

FAST VP allocation by FAST Policy A new feature for FAST VP in 5876 is the ability for a device to allocate new extents from any thin pool participating in the FAST VP Policy. When this feature is enabled, FAST VP will attempt to allocate new extents in the most appropriate tier, based upon performance metrics. If those performance metrics are unavailable it will default to allocating in the pool to which the device is bound. If, however, the chosen pool is full, regardless of performance metrics, then FAST VP will allocate from one of the other thin pools in the policy. As long as there is space available in one of the thin pools, new extent allocations will be successful.

This new feature is set at the Symmetrix array level and applies to all devices managed by FAST VP. The feature cannot, therefore, be applied to some FAST VP policies and not others. By default it is disabled and any new allocations will come from the pool to which the device is bound.

A pinned device is not considered to have performance metrics available and therefore new allocations will be done in the pool to which the device is bound.

Oracle Applications Oracle Applications, or Oracle E-Business Suite, is a tightly integrated family of Financial, ERP, CRM, and manufacturing application products that share a common


look and feel. Using the menus and windows of Oracle Applications, users have access to all the functions they need to manage their business information. Oracle Applications is highly responsive to users, supporting a multi-window GUI that provides users with full point-and-click capability. In addition, Oracle Applications offers many other features such as field-to-field validation and a list of values to help users simplify data entry and maintain the integrity of the data they enter.

VPLEX EMC VPLEX is unique virtual storage technology that federates data located on multiple storage systems – EMC and non-EMC – allowing the storage resources in multiple data centers to be pooled together and accessed anywhere. When combined with virtual servers, it is a critical enabler of private and hybrid cloud computing and the delivery of IT as a flexible, efficient, and reliable resilient service.

The VPLEX family addresses three primary IT needs:

Mobility: The ability to move applications and data across different storage installations, whether within the same data center, across a campus, within a geographical region – and now, with VPLEX Geo, across even greater distances.

Availability: The ability to create high-availability storage infrastructure across these same varied geographies with unmatched resiliency.

Collaboration: The ability to provide efficient real-time data collaboration over distance for such big data applications as video, geographic/ oceanographic research, and others.

All of this can be done within or across data centers, located synchronous or asynchronous distances apart, in a heterogeneous environment.

The VPLEX family brings many unique innovations and advantages:

VPLEX technology enables new models of application and data mobility, leveraging distributed virtual storage. For example, VPLEX is specifically optimized for virtual server platforms (e.g., VMware ESXi, Hyper-V, Oracle Virtual Machine, and AIX VIOS) and can streamline and even accelerate transparent workload relocation over distances, which includes moving virtual machines over distances.

With its unique, highly available, scale-out clustered architecture, VPLEX can be configured with one, two, or four engines – and engines can be added to a VPLEX cluster non-disruptively. All virtual volumes presented by VPLEX are always accessible from every engine in a VPLEX cluster. Similarly, all physical storage connected to VPLEX is accessible from every engine in the VPLEX cluster. Combined, this scale-out architecture uniquely ensures maximum availability, fault tolerance, and scalable performance.


Advanced data collaboration, through AccessAnywhere, provides cache-consistent active-active access to data across two VPLEX clusters over synchronous distances with VPLEX Metro and asynchronous distances with VPLEX Geo.

Mobility

Application and data mobility provides the movement of virtual machines (VM) without downtime. Storage administrators have the ability to automatically balance loads through VPLEX, using storage and compute resources from either cluster’s location. When combined with server virtualization, VPLEX can transparently move and relocate virtual machines and their corresponding applications and data over distance. This provides a unique capability to relocate, share, and balance infrastructure resources between sites, which can be within a campus or between datacenters, up to 5 ms round trip time (RTT) latency apart with VPLEX Metro, or further apart (50 ms RTT) across asynchronous distances with VPLEX Geo.

Availability

By providing redundancy, flexibility, and awareness (through VPLEX Witness), GeoSynchrony supports small recovery time objective (RTO) and recovery point objective (RPO).

All of these features allow the highest resiliency possible in the case of an outage.

VPLEX Product Offerings

VPLEX first meets high-availability and data mobility requirements and then scales up to the I/O throughput you require for the front-end applications and back-end storage.

The three available VPLEX product offerings in Figure 11 along with an optional RecoverPoint configuration are:

VPLEX Local

VPLEX Metro

VPLEX Geo


Figure 11. VPLEX configurations

A VPLEX cluster consists of a single-engine, dual-engines, or quad-engines and a management server. Each engine contains two directors. A dual-engine or quad-engine cluster also contains a pair of Fibre Channel switches for communication between directors and a pair of UPS (Uninterruptible Power Sources) for battery power backup of the Fibre Channel switches and the management server.

The management server has a public Ethernet port, which provides cluster management services when connected to the customer network.

VPLEX Local provides seamless, non-disruptive data mobility and the ability to manage and mirror data between multiple heterogeneous arrays from a single interface within a data center. VPLEX Local consists of a single VPLEX cluster. VPLEX Local is a next-generation architecture that allows increased availability, simplified management, and improved utilization and availability across multiple arrays.

VPLEX Metro enables active/active, block level access to data between two sites within synchronous distances. The distance is limited not only by physical distance but also by host and application requirements. Depending on the application, VPLEX clusters should be installed with inter-cluster links that can support not more than 5ms round trip time (RTT).

The combination of virtual storage with VPLEX Metro and virtual servers enables the transparent movement of virtual machines and storage across synchronous instances. This technology provides improved utilization and availability across heterogeneous arrays and multiple sites.


VPLEX Geo enables active/active, block level access to data between two sites within asynchronous distances. VPLEX Geo enables more cost-effective use of resources and power.

VPLEX Geo extends the distance for distributed devices up to and within 50ms RTT. As with any asynchronous transport media, you must also consider bandwidth to ensure optimal performance. Due to the asynchronous nature of distributed writes, VPLEX Geo has different availability and performance characteristics than Metro.

For all VPLEX products, GeoSynchrony:

Presents storage volumes from back-end arrays to VPLEX engines

Federates the storage volumes into hierarchies of VPLEX virtual volumes with user-defined configuration and protection levels

Presents virtual volumes to production hosts in the SAN via the VPLEX front-end

For VPLEX Metro and VPLEX Geo products, presents a global, block-level directory for distributed cache and I/O between VPLEX clusters

Location and distance determine high-availability and data mobility requirements.

When back-end storage arrays or application hosts span two data centers, the AccessAnywhere feature in VPLEX Metro or a VPLEX Geo federates storage in an active/active configuration between VPLEX clusters. Choosing between VPLEX Metro and VPLEX Geo depends on distance, availability, and data synchronicity requirements.

Application and back-end storage I/O throughput, along with availability requirements determine the number of engines in each VPLEX cluster.

High-availability features within the VPLEX cluster allow for non-disruptive software upgrades and hardware expansion as I/O throughput increases.

Logging volumes

Logging volumes are used to keep track of blocks written during an inter-cluster link failure or when one leg of a distributed RAID-1 becomes unreachable and then recovers. After a link failure is restored or an unreachable leg recovers, VPLEX uses the information in logging volumes to synchronize the mirrors by sending only changed blocks across the link.

Redundancy with RecoverPoint

EMC RecoverPoint provides comprehensive data protection by continuous replication (splitting) of host writes. With RecoverPoint, applications can be recovered to any point in time. Replicated writes can be written to local volumes to provide recovery from operational disasters, to remote volumes to provide recovery from site disasters, or both.

RecoverPoint supports 3 types of splitters:


Host OS-based splitters

Intelligent fabric-based splitters (SANTap and Brocade)

Storage-based splitters (CLARiiON CX4, VNX series, and Symmetrix VMAX)

Starting in GeoSynchrony 5.1, VPLEX includes a storage-based RecoverPoint splitter. The splitter is built into VPLEX such that VPLEX volumes can have their I/O replicated by RecoverPoint appliances to volumes located in VPLEX or on one or more heterogeneous storage arrays.

For GeoSynchrony 5.1, RecoverPoint integration is offered for VPLEX Local and VPLEX Metro configurations (not for Geo).

The VPLEX splitter enables VPLEX volumes in a VPLEX Local or VPLEX Metro to mirror I/O to a RecoverPoint appliance performing continuous data protection (CDP), continuous remote replication (CRR), or concurrent local and remote data protection (CLR).

VASA with VPLEX

VMware vStorage API for Storage Awareness or VASA is available with VPLEX arrays and can be configured from within VMware vCenter. The VPLEX VASA provider is delivered as a vApp. Figure 12 shows the configuration from within the vCenter

Figure 12. VPLEX VASA Provider registration


The VASA provider includes storage capabilities that can be used to categorize virtual machines, datastores, etc. The default ones that come with the provider are shown in Figure 13.

Figure 13. VPLEX VASA storage capabilities

The details around implementing VASA are beyond the scope of this white paper. Please use the VPLEX Procedure Generator for more detail which can be found on Support.EMC.com.

FAST VP and Oracle Applications 12

Applications Architecture

The Oracle Applications Architecture is a framework for multi-tiered, distributed computing that supports Oracle Applications products. In this model, various servers or services are distributed among three levels, or tiers.

A tier is a logical grouping of services, potentially spread across more than one physical or virtual machine. The three-tier architecture that comprises an Oracle E-Business Suite installation is made up of the database tier, which supports and manages the Oracle database; the application tier, which supports and manages the various Applications components, and is sometimes known as the middle tier; and the desktop tier, which provides the user interface through an add-on component to a standard web browser.

The simplest architecture for Oracle Applications is to have all tiers, except the desktop tier, installed on a single server. This configuration might be acceptable in a development environment, but for production environments scaling would quickly become an issue. In order to mimic a more realistic production environment, therefore, the architecture of the testing environment is built with two physical


application tiers and two database tiers. The architectural components of Oracle Applications are seen in Figure 14.

Figure 14. Oracle Applications architecture

Working with FAST VP and Oracle Applications on VMware vSphere

Because of the diversity of Oracle Applications, in that there are hundreds of different modules within a single product, deploying them appropriately on the right tier of storage is a daunting task. Implementing them in a VMware environment that utilizes both VPLEX and then FAST VP on the underlying array will demonstrate how a customer can achieve proper performance and cost savings at the same time. For this paper, the latest Oracle Applications release 12 was installed and configured. There are a few benefits to using the latest release. First, Oracle pre-packages release 12 with version 11g of the Oracle database. Version 11g is Oracle’s latest database release and represents a significant advancement over 10g in performance and functionality. Secondly, Oracle has now divorced itself from the practice of having two tablespaces, and hence at least two datafiles, per application. Prior to release 12, each Applications module had its own set of tablespaces and datafiles, one for the data and one for the index. With over 200 schemas, managing a database of over


400 tablespaces and datafiles was, and is, a sizable undertaking. The new approach that Oracle uses in release 12 is called the Oracle Applications Tablespace Model, or OATM.

Oracle Applications Tablespace Model

Oracle Applications release 12 utilizes as the standard a modern infrastructure for tablespace management, the Oracle Applications Tablespace Model (OATM). The OATM is similar to the traditional model in retaining the system, undo, and temporary tablespaces. The key difference is that Applications products in an OATM environment share a much smaller number of tablespaces, rather than having their own dedicated tablespaces.

Applications schema objects are allocated to the shared tablespaces based on two main factors: the type of data they contain, and I/O characteristics such as size, life span, access methods, and locking granularity. For example, tables that contain seed data are allocated to a different tablespace from the tables that contain transactional data. In addition, while most indexes are held in the same tablespace as the base table, indexes on transaction tables are held in a single tablespace dedicated to such indexes.

Oracle Applications implementation

A customer implementation of Oracle Applications is not a quick process. The installation itself is only the first part of what can be an endeavor lasting many months or longer. Although there are almost 200 application modules in the Oracle Applications, customers rarely, if ever, use all of them. They use a selection of them, or perhaps a bundle such as Financials, or CRM. These modules are then implemented (typically) in a phased approach. The transition from an existing applications system or implementation of a new system takes time. How that system eventually will be used, and more importantly how that database will be accessed, presents a real challenge for the system administrator and database administrator. These individuals are tasked with providing the right performance for the right application at the right price. In other words, both performance optimization and cost optimization are extremely important to them; however, obtaining the balance between the two is not an easy task. The three disk technologies utilized on the arrays backing the VPLEX in this environment — SATA, FC, and EFD — have differing performance characteristics and very different costs. For instance, how will an administrator decide what part of the database belongs on the various disk technologies that represent the cost and performance balancing act they attempt each day? This is made even more difficult under the new Oracle Applications Tablespace Model. Oracle’s new model certainly does a good job at high-level database object consolidation – less tablespaces and less datafiles – but since the application modules are no longer separated into individual tablespaces and datafiles, there really is no practical way to put different modules on different tiers of storage – until now. FAST VP is the perfect complement to the manner in which Oracle implements the database in release 12 of the Oracle application suite. In fact, the entire database of user data can be placed on a single mount point on a VMware


virtual disk and yet still be spread over the appropriate disk technologies that match the business requirements. The simplicity of this deployment model is enabled by FAST VP.

For more detail about implementing Oracle Applications in a FAST VP enabled environment please see the white paper Storage Tiering for VMware Environments Deployed on EMC SYMMETRIX VMAX with Enginuity 5876 on EMC.com. The use case presented in that document is similar to the FAST VP implementation in this white paper.

Oracle Applications deployment

The Oracle Applications’ VMware environment deployed in this study consists of two ESXi 5.1 servers with a total of four virtual machines listed in Table 1. The environment is managed by a VMware vCenter Server. Figure 15 is a visual representation of the environment.

Table 1. Example environment

Virtual Machine Name Model OS &

Version

CPUs RAM

(GB)

Disk

RAC Database Tier

Node 1

dsib2124 VMware

VM

OEL 5

64-bit

4 8 SAN

RAC Database Tier

Node 2

dsib2125 VMware

VM

OEL 5

64-bit

4 8 SAN

Applications Tier 1 dsib2126 VMware

VM

OEL 5

64-bit

1 2 SAN

Applications Tier 2 dsib2127 VMware

VM

OEL 5

64-bit

1 2 SAN

Production vCenter dsib2113 VMware

VM

Win2008

64-bit

2 4 SAN

Recovery vCenter dsib2114 VMware

VM

Win2008

64-bit

2 4 SAN

VPLEX Metro

Witness

dsib2013 vApp Custom

Linux

1 .5 SAN

VPLEX VASA dsib2111 vApp Custom

Linux

1 3.6 SAN

EMC Symmetrix

VMAX

HK198700067 VMAX 10K 5876.229.145

microcode

VPLEX cluster-1 SATA,FC,

EFD

EMC Symmetrix

VMAX

HK198700068 VMAX 10K 5876.229.145

microcode

VPLEX cluster-2

w/ RP Splitter

SATA,FC,

EFD

EMC Symmetrix

VMAX

HK195900283 VMAX 10K 5876.229.145

microcode

RP Remote Site

w/RP Splitter

SATA,FC,

EFD


Hardware layout

Figure 15. Physical/virtual environment diagram

FAST VP in the VPLEX Metro Cluster As the VPLEX Metro cluster is backed by VMAX arrays, and the remote site with RecoverPoint is also a VMAX array, the environment takes advantage of FAST VP. Each array has three disk technologies, EFD, FC, and SATA. The disks are configured as follows:

EFD – RAID-5 3+1

FC – RAID 1, or 2-way mirror

SATA – RAID-6 (6+2)

The screen capture in Figure 16 shows this:


Figure 16. Thin Pool configuration for VMAX array behind VPLEX

For the two arrays in the VPLEX Metro configuration, the enabled capacity in the pools is about the same. For the DR site, the pools are larger due to the fact it serves as a test and development bed while DR is not activated. The general settings for FAST VP are seen in Figure 17.

Figure 17. FAST VP system settings

Most of the settings were left as default, with a few notable exceptions. First, Allocation by FAST Policy was used. This feature was explained previously and not only helps place the new extent in the appropriate tier, but if the binding pool is full it will use another. The second exception is for compression, as the thin pools involved


in the FAST VP policy were enabled for compression. Enginuity version 5876.159.102 introduced compression capabilities for Virtually Provisioned environments (VP compression) that provides customers with 2:1 or greater data compression for very infrequently accessed data3. FAST VP manages the compression automatically. Note that for this environment, the maximum amount of days has been set (400) since the financial data will require monthly, quarterly, and year-end reports to be run. Running those reports against compressed data would be far too costly in terms of processing power and time.

All the other disks of the database become part of the FAST VP Policy, named EBS_Policy. In this policy the RecoverPoint journals are also assigned. RecoverPoint journals will benefit from the same set of disk technologies as our database. The policy percentages of disk technologies used in this use case speak to two realities: first that both EFD and FC are more expensive mediums than SATA, and second, and more importantly, that only a small percentage of an application or database is going to be accessed regularly. FAST is designed to make use of the disk provided to it. If cost were not a concern, the best policy to institute is 100/100/100, which would allow FAST full reign to use as much of each tier as it needed. Unfortunately, in the real world cost is one of the prime concerns, and as a result customers are more likely to have smaller amounts of FC and EFD in their Symmetrix than SATA. This leaves less to dedicate to a FAST VP policy; however the good news is that most data in applications and databases are rarely accessed so it is unlikely large amounts of very fast disks such as EFD will be required. As this is a RecoverPoint implementation, the amount of space needed for the journals must also be considered.

For each array Figure 18 shows the percentages used for the policy are: EFD 10%, FC 50% and SATA 100%.

Figure 18. FAST VP policy for Oracle EBS environment

FAST VP policies need to be associated with storage groups in order to operate on devices. More than one storage group may be assigned to the same policy, as will be the case in this environment, or multiple policies can be created for each storage

3 On data that can actually be compressed by the compression engine.


group. There are a number of ways to assign the storage groups to the FAST VP policy. In Figure 19 the Storage Groups screen in Unisphere for VMAX is utilized. The four step process displayed is:

1. Select the storage group

2. Select the “Associate to FAST” button

3. Choose the FAST VP Policy to associate

4. Select “OK” to complete

Figure 19. Associating a FAST VP policy to a storage group

This process was repeated on each array for the two storage groups used on each. Figure 20 and Figure 21 show the FAST VP setup of each array that is part of the VPLEX Metro cluster. Note that on each array the storage groups are in compliance, meaning there is enough thin pool space in each of the tiers to satisfy the current extent allocation on the devices.


Figure 20. FAST VP overview of EBS_Policy on VMAX behind cluster-1 of VPLEX


Figure 21. FAST VP overview of EBS_Policy on VMAX behind cluster-2 of VPLEX

Although not explicitly seen in Figure 21, the storage groups on the cluster-2 array have a greater capacity requirement than those on the cluster-1 array. The reason for this is that cluster-2 is also the local RecoverPoint site. That being the case, there are journal devices for each corresponding device in the distributed relationship with the cluster-1 array. This is important to remember when configuring a customer environment, that being the storage requirements will be greater for the VPLEX Metro cluster that is serving the dual-purpose of the local RecoverPoint implementation. If the cluster arrays are similar in configuration it may be that one or more storage groups will be non-compliant when first associated with a FAST VP policy. This is not in and of itself a bad thing, for there are many extents that can be relocated to the SATA tier where the policy holds that 100% of the devices may reside. The storage groups therefore will come into compliance within a short time after FAST’s initial analysis period.

RecoverPoint Remote Site

Just as the two arrays associated with VPLEX Metro will employ FAST VP for the environment, the disaster recovery site, a VMAX 10K, will also. The VMAX 10K used in


the configuration has the benefit of a larger FC pool than either arrays in the VPLEX configuration and therefore even with the extra storage commanded by the journals, the storage group is in compliance immediately upon association with the FAST VP policy. Note that the DR site seen in Figure 22 uses a single storage group, unlike the production environment. This is just a matter of choice. The replicas and the journals can be separated into different storage groups and then each group associated with the same FAST VP policy. The capacity requirements, however, would be the same.

Generally the journals at the DR site will remain at the higher performing disk such as FC because they are written to heavily. The replica is less so, but if DR is invoked, FAST VP will move the now production extents to the higher performing tiers as the access patterns change.

Figure 22. FAST VP overview of EBS_Policy on VMAX at 3rd site with RecoverPoint

Device Pinning in FAST VP for the Oracle Database When a FAST VP policy is associated with a storage group, all the devices in that storage group are also. It may be necessary or desirable, however, to prevent FAST VP from moving extents of some devices in the storage group. For example, if there is a device in the group which will be used to back a VMFS datastore for ISO images, and


it has been bound to a SATA pool, there would be no reason for extents of that device to be promoted. Now granted, while accessed for its intended use, there is little chance FAST VP would ever up-tier extents from the device, but accidental access to the datastore for an intensive I/O VM could. In this case it may be best to simply “pin” the device. Pinning will prevent FAST VP from moving any extents of the device and thus keep it in SATA.

In the VPLEX Metro environment only two devices were pinned from each array, the REDO devices of the database. These will always require FC, RAID 1 capabilities so we will leave those in the FC pool. SATA would be too slow while EFD would not be cost-effective given the heavy write activity of the logs.

Pinning the REDO devices in FAST VP

Pinning a device can be done in Unisphere for VMAX or through the SYMCLI. In Unisphere, shown in Figure 23, one navigates through the menu path: array -> Storage -> Storage Groups to pin a REDO disk. The user then highlights the device and clicks the double-arrow icon on the bottom menu. From the pop-up menu, the user selects “Pin”. By pinning the device there is no concern that the FAST engine will move extent groups belonging to that device. Note that multiple devices may be selected at once for pinning.


Figure 23. Pinning a device in Unisphere for VMAX

VPLEX Distributed Devices and Consistency Groups The volumes used in the environment are all comprised of distributed devices which have been incorporated into consistency groups. This is required for integration with RecoverPoint, though it is always a best practice to use consistency groups. The consistency groups are constructed around Oracle RAC and the Oracle Applications E-Business Suite.


Provisioning Storage example

This section will walk through the creation of a distributed device, its consistency group, and then presenting it to both an ESXi host and the RecoverPoint appliances. The example demonstrates the same process that was repeated for all distributed devices required in the Oracle E-Business Suite Extended RAC environment. The interface used for this example is Unisphere for VPLEX.

As the process of creating a distributed device relies on a storage device from each cluster of a VPLEX Metro and one virtual volume, only one cluster will be used in the example. A duplicate process would need to be conducted on the other cluster in the VPLEX Metro. Note that as only one virtual volume is needed between the two clusters, one of the virtual volumes will be removed from one of the clusters, leaving just the storage device. The process of removing the virtual volume will be outlined below.

The GUI interface in Unisphere for VPLEX provides for a simplified wizard that will take the user through claiming the storage all the way to the virtual volume creation. There is of course the capability to take a step-by-step approach when creating the virtual volume in case there are specific needs such as adjusting the extent size or to change the RAID configuration if say a Symmetrix VMAX or EMC array were not being used behind the VPLEX. Fortunately when using advanced arrays like VMAX, the default selections are the correct ones and hence the simplified wizard is presented.

We start the process by rediscovering the array in Figure 24. This will force a rescan on the HBAs to determine if any new storage has been presented from the VMAX to the VPLEX.


Figure 24. Provision Storage – Rediscover array for new volumes

VPLEX Virtual Volume Wizard

Next the storage that was discovered needs to be claimed and then a virtual volume built. This is done through a single-button wizard. From the same screen the selection “Create Virtual Volumes” is made in Figure 25.


Figure 25. Provision Storage – Create Virtual Volumes

The first screen calls for deciding between using unclaimed or claimed storage. As the storage was just presented from the VMAX, the storage is unclaimed and the radio button is select to reflect that in Figure 26.


Figure 26. Provision Storage – Claim unclaimed storage

In the next step for the VMAX, as one can see in the highlighted sentence in Figure 27, we do not need to be concerned about a mapping file. We do wish to use thin rebuilds because all our devices are provisioned on a VMAX 10K, a completely thin array. VPLEX does not automatically detect if a storage volume is thinly-provisioned on the array. When you select a thin rebuild, VPLEX preserves the thinness of the target device by copying only non-zero data.

Figure 27. Provision Storage – Use thin rebuilds

From the screen in Figure 28 we then select the single device that the VPLEX discovered and ask it to claim it.


Figure 28. Provision Storage – Claim storage

In the final screen in Figure 29 we will be presented with the option of naming the virtual volume. By default VPLEX will assign a name that uses both the array ID and the device ID from the array. The name can be customized to anything that is desired; however, using an identifier that ties it to the cluster on which the virtual volume is being created, may be useful in later troubleshooting exercises or in other tools that present the name to the user.4 In the following example, for this reason, the name used is “VPLEX_DEMO_cluster-1.”

4 As noted, the system names that are generated default to a combination of the array ID and the device ID. While initially this may be useful in distinguishing between backing storage arrays, tech refreshes may change arrays completely and the array information would not be relevant. It may make sense, therefore, to use names related to hosts or applications for storage objects, or in this case by cluster.


Figure 29. Provision Storage – Creating virtual volume

The final results are presented in Figure 30, both for cluster-1 and the corresponding virtual volume for the distributed device on cluster-2.


Figure 30. Provision Storage – Virtual volumes VPLEX_DEMO_cluster-1 and VPLEX_DEMO_cluster-2

Once the virtual volumes are in place on each cluster, we need to remove the virtual volume on cluster-2 as recall only one virtual volume is needed for the distributed device. As we have an existing virtual volume on cluster-1, we are going to add a remote mirror to it.5 The removal of the virtual volume is very straightforward through the wizard. Start by selecting the virtual volume “VPLEX_DEMO_cluster-2” in the Virtual Volumes view for cluster-2 and select “Delete” as seen in Figure 31.

5 For a complete explanation of the relationship between VPLEX virtual volumes, devices, extents, and storage volumes, select the Provision Storage menu then the Provisioning Overview option in Unisphere for VPLEX.


Figure 31. Provision Storage – Virtual volume deletion

Next, confirm that you wish to delete the virtual volume and then close the summary screen, both seen in Figure 31. Deleting the virtual volume only removes that object and leaves the underlying storage device. This device can now be used as the remote mirror for the virtual volume and underlying storage device on cluster-1. After deletion the user may wish to rename the storage device to a similar format as used in the virtual volume creation. In this example the storage device on cluster-2 was renamed in Unisphere to “device_VPLEX_DEMO_cluster-2”.


Figure 32. Provision Storage – Confirm virtual volume deletion

With the virtual volume at cluster-1 and the corresponding device at cluster-2 we are ready to add a remote mirror. In adding a remote mirror on cluster-2, the virtual volume from cluster-1 becomes available across both clusters. This is because the device underneath the virtual volume is now a distributed device.

The same naming convention was used for the virtual volume on cluster-1 and the storage device on cluster-2: VPLEX_DEMO_cluster-1 on cluster-1 and device_VPLEX_DEMO_cluster-2 on cluster-2.

Note that the distributed device creation will fail if the device on each cluster has the same name, or if the remote device is smaller than the device being mirrored. Unisphere for VPLEX will provide no warning of this until the completion of the wizard and the attempt at creation so be sure each device name is unique and that the remote device is equal to or greater than the device being mirrored.

VPLEX Distributed Device Creation

While still in the Provisioning wizard, in the menu on the left, select cluster-1 and then Virtual Volumes. Highlight the virtual volume on cluster-1, VPLEX_DEMO_cluster-1. Select “Add Remote Mirror” to start the process of creating the distributed device. This process is show in Figure 33.


Figure 33. Distributed Devices – Add Remote Mirror

In the following screen the virtual volume is already pre-selected so simply hit “Next” to continue. This is shown in Figure 34.

Figure 34. Distributed Devices – Add virtual volume


As there is only a single device of appropriate size configured on cluster-2 to match with, the mirror mapping shown in Figure 35 is simple. Obviously this is more often the exception than the rule and why the naming conventions are important for the devices from each cluster. There may be many devices of the same size and using only system names would complicate what should be a fairly straightforward process. After highlighting the device from cluster-1 and the device from cluster-2, select the “Add Mirror” button and then click “Next” or “Finish”.

Figure 35. Distributed Devices – Select remote mirror for distributed device

In the subsequent screen, Figure 36, we now may choose to add this distributed device to an existing consistency group, or create a new one. We’ll choose to create a new one and name it the same as our base name for our local devices, VPLEX_DEMO. In addition to the consistency group a rule set must be applied to the consistency group.

Rule Set and RecoverPoint

The rule set defines which VPLEX cluster will continue I/O in the event of a dual WAN partition between VPLEX clusters. The winning site will continue servicing host I/O. Because we are using RecoverPoint in our implementation, we must choose the site where RecoverPoint is configured, or cluster-2, though be aware that VPLEX will not


prevent the user from selecting one of the other options.6 This is because if there is a WAN partition we want to be sure that the volumes under RecoverPoint protection are the ones who continue to receive the I/O. If cluster-1 were the winner, RecoverPoint would not be replicating the changes and thus the DR site would represent a point-in-time of the WAN failure.

Figure 36. Distributed Devices – Create consistency group

The final screen in Figure 37 summarizes everything. The results screen in Figure 38 shows whether or not the creation was successful.

6 Fortunately, even if the wrong rule set is made, once the consistency group is enabled for RecoverPoint, VPLEX will force the correct rule set and prevent the user from modifying it.


Figure 37. Distributed Devices – Virtual volume name

Figure 38. Distributed Devices – Review results

Failure status of distributed device

Note that while the devices are synchronized, Unisphere will report failures on the distributed device because it is rebuilding.7 This status is highlighted in Figure 39. Once the Health and Status are “OK” the remaining steps to make this device usable in the environment can be completed.

7 By default, VPLEX will synchronize the storage devices on each cluster. If the array devices are being newly provisioned to the VPLEX clusters, it may not be desirable to synchronize the devices as there is no data on them and the time to do so could be significant if the devices are large. If the user wishes not to synchronize, the storage components need to be built individual one after the other: storage volume, extent, device, etc., as opposed to using the single wizard for virtual volume creation. Using this methodology the user will be presented with an option of whether to synchronize at the appropriate point. EMC best practice, however, is to synchronize the devices.


Figure 39. Distributed Devices – Rebuilding status

The speed of the rebuild can be controlled by adjusting the max-transfer size parameter on each distributed device. The default is 128 KB and can be increased depending on the synchronization speed needed. If the virtual volumes are not yet being used, it is safe to increase this to 2 MB to speed up the process for larger virtual volumes. Please see the VPLEX documentation for more information on how to do this.

Enabling RecoverPoint on the VPLEX consistency group

Assuming that RecoverPoint has been configured on VPLEX following the product documentation, when the rebuild is complete we want to enable RecoverPoint on the consistency group before placing it in the correct storage views. Select the consistency group on which to enable RecoverPoint. This will bring up the dialog box which has the option to “Enable RecoverPoint Usage” as a checkbox. Check the box in Figure 40 then select “OK”.


Figure 40. Consistency Group – Enable RecoverPoint

Adding distributed devices to storage views

With RecoverPoint enabled, we will now put the distributed device in three separate storage views. The first view is the one that will present the storage to the ESXi host which is attached to cluster-1 of our Extended RAC environment, and the second to the other host which is zoned to cluster-2. The third view is only on cluster-2 and is the view that presents storage to the RecoverPoint appliances. In order to use this distributed device with RecoverPoint, RecoverPoint must be able to see them.

The first figure, Figure 41, shows how to add the distributed device to the DSIB1131_view in a four step process:

1. Select Add/Remove Virtual Volumes

2. Select the volume from the left-hand column and then “Add” to move it over to the volumes already presented.

3. Select “OK” to complete adding the volume.

4. Close the results.


Figure 41. Storage Views – Adding a volume

Adding the volume to the second view requires an additional step. When we go to add this distributed device to the view DSIB1132_view on cluster-2, it initially cannot be seen in the left-hand column. This is because Unisphere will hide any devices that are already in a view. It is necessary to check the box highlighted in Figure 42 and allow those hidden volumes to be shown as seen in the inset.


Figure 42. Storage Views – Multiple views

Even though the box is checked, Unisphere is going to warn the user in Figure 43 about adding a device to a view when it is already in another view:

Figure 43. Storage Views – Warning


It will warn again in the summary in Figure 44. This environment is, however, one of those very specific circumstances the warning speaks about. Note that the same set of events will occur again when the device is added to the view for RecoverPoint.

Figure 44. Storage Views – Summary

Viewing in VMware

As the storage is mapped and masked to the ESXi host it can be discovered in the vSphere Client. Run a rescan of the HBAs as in Figure 45 to recognize the new VPLEX distributed device.


Figure 45. vSphere Rescan for Datastores

Now use EMC VSI Storage Viewer (Figure 46) to see all the details of the distributed device:


Figure 46. VSI Storage Viewer – VPLEX distributed device

At this point the device is now ready to be used in the Oracle E-Business Suite Extended RAC Cluster and then subsequently configured with RecoverPoint to the third site. The last step will be to configure the VMware SRM environment. Note that the steps need not follow this exact path. One option might be to create the entire VPLEX VMware environment first before adding the devices to the RecoverPoint views and enabling VMware SRM. It is up to the customer to decide what will best work in their environment.

VPLEX Local Devices for RecoverPoint Journals For VPLEX volumes that will be used as RecoverPoint journals the process is different than the one just described for distributed devices since only a single virtual volume is used. Recall that RecoverPoint is only attached to one of the clusters in the VPLEX Metro. Therefore distributed devices will not be used, just regular virtual volumes. To create the virtual volumes we follow the same procedure as for distributed devices, but only on a single cluster. The process was detailed in the section Provisioning Storage example. From here we will create a consistency group and present it to the RecoverPoint appliance.

Note that RecoverPoint can only provide protection for a given virtual volume at one of the clusters. Therefore, the journal should be located at the site where the RPA providing the splitting resides. We do not use a distributed device for the journal as it cannot be used by an RPA at the other cluster, even if there were a splitter. This is due to the fact that while a VPLEX Metro can have RecoverPoint clusters attached and splitting at both clusters, each virtual volume may only be protected by RecoverPoint at one VPLEX cluster at a time. Please note that the RecoverPoint journal can also be placed directly onto physical storage and presented to the RPA rather than through VPLEX. This may make sense when the goal is to minimize the impact of journal I/O on the VPLEX engines, or if the current engines are already at or near peak utilization prior to the introduction of RecoverPoint. The best practice recommendation,


however, is to place the journal on a VPLEX volume so that a tech refresh on the underlying array is non-disruptive to RecoverPoint replication.

Consistency Group creation for RecoverPoint Journal

For these virtual volumes designated as RecoverPoint journals it is still necessary to create a consistency group, or add them to an existing group. As there are separate consistency groups for each of the RecoverPoint groups, we will create a separate RecoverPoint group for the journals which will also contain the RecoverPoint repository device. Having those devices in the same consistency group would be the best practice.

Use the menu under Virtualized Storage to select Consistency Groups. Then select “Create” and fill in the consistency group name. By default, the local cluster will be selected to designate from where the virtual volumes should come. Note the check box for global visibility. This should remain unchecked as none of these volumes should be made available to any cluster but cluster-2. Figure 47 has the detail of the steps. Select “Next” or “Finish” when complete.

Figure 47. Create local consistency group

In the next set of screenshots in Figure 48 follow the three steps:

1. Add the virtual volume from the right-hand column to the left-hand column and select “Next”.


2. Commit the creation in the final screen.

3. Review the summary screen and close.

Note in this example the virtual volume name is “VPLEX_DEMO_1_vol” so as not to confuse it with our previous examples.

Figure 48. Add VPLEX local volume to consistency group


Adding RecoverPoint Journal to a storage view

The devices can now be added to the storage view. Also unlike the distributed devices, the journals will only be added to a single view and moreover on only the single cluster attached to the RecoverPoint splitter. These are volumes used by RecoverPoint and will never be accessed (and cannot be) by a host. Figure 49 walks the user through adding the volume to the view. It is the same process explained previously in Figure 41.

Figure 49. Adding local volume to storage view


VMware Implementation

VMware deployments in a VPLEX Metro environment

EMC VPLEX breaks physical barriers of data centers and allows users to access data at different geographical locations concurrently. This functionality in a VMware context enables functionality that was not available previously. Specifically, the ability to concurrently access the same set of devices independent of the physical location enables geographically stretched clusters based on VMware vSphere8. This allows for transparent load sharing between multiple sites while providing the flexibility of migrating workloads between sites in anticipation of planned events such as hardware maintenance. Furthermore, in case of an unplanned event that causes disruption of services at one of the data centers, the failed services can be quickly and easily restarted at the surviving site with minimal effort. Nevertheless, the design of the VMware environment has to account for a number of potential failure scenarios and mitigate the risk for services disruption. The following paragraphs discuss the best practices for designing the VMware environment to ensure an optimal solution. For further information on EMC VPLEX Metro configuration readers should consult the TechBook EMC VPLEX Metro Witness Technology and High Availability available on Support.EMC.com.

VPLEX Metro Witness

VPLEX uses rule sets to define how a site or link failure should be handled in a VPLEX Metro or VPLEX Geo configuration. If two clusters lose contact, the rule set defines which cluster continues operation and which suspends I/O. The rule set is applied on a device-by-device basis or in this case for a consistency group. The use of rule sets to control which site is a winner, however, adds unnecessary complexity in case of a site failure since it may be necessary to manually intervene to resume I/O to the surviving site. VPLEX with GeoSynchrony 5.0 introduced a concept to handle such an event, the VPLEX Witness. VPLEX Witness is delivered as a zero cost VMware Virtual Appliance (vApp) which runs on a customer supplied ESXi server. The ESXi server resides in a physically separate failure domain to either VPLEX cluster and uses different storage to the VPLEX cluster.

It provides the following features:

Active/active use of both data centers

High availability for applications (no single points of storage failure, auto-restart)

Fully automatic failure handling

Better resource utilization

Lower capital expenditures and lower operational expenditures as a result

8 The solution requires extension of VLAN to different physical data centers. Technologies such as Cisco’s Overlay Transport Virtualization (OTV) can be leveraged to provide the service.


As one can see from Figure 50 VPLEX Witness converts a VPLEX Metro from an active/active mobility and collaboration solution into an active/active continuously available storage cluster. We can see that the Witness VM is deployed in a separate fault domain (as defined by the customer) and connected into both VPLEX management stations via an IP network. Furthermore once VPLEX Witness is deployed, failure scenarios become self-managing (i.e. fully automatic) which makes it extremely simple since there is nothing to do regardless of failure condition.

Figure 50. VPLEX Witness

Note: Fault domain is decided by the customer and can range from different racks in the same datacenter all the way up to VPLEX clusters 5ms of distance away from each other (5ms measured latency or typical synchronous distance).

A configuration that uses any combination of VPLEX Metro and VPLEX Witness allows both clusters to provide coherent read/write access to the same virtual volume. That means that on the remote site, the paths are up and the storage is available even before any failover happens. When this is combined with host failover clustering technologies such as VMware HA, one gets a fully automatic application restart for any site-level disaster. The system rides through component failures within a site, including the failure of an entire array.

VMware ESXi can be deployed at both VPLEX clusters in a Metro environment to create a high availability environment. Figure 51 shows the Metro HA configuration that is used in this paper.


Figure 51. VMware Metro HA with VPLEX Witness

In this scenario, a virtual machine can write to the same distributed device from either cluster. In other words, if the customer is using VMware Distributed Resource Scheduler (DRS), which allows the automatic load distribution on virtual machines across multiple ESXi servers, a virtual machine can be moved from an ESXi server attached to cluster-1 to an ESXi server attached to cluster-2 without losing access to the underlying storage. This configuration allows virtual machines to move between two geographically disparate locations with up to 5 ms of latency, the limit to which VMware vMotion is supported.

In the event of a complete site failure, VPLEX Witness automatically signals the surviving cluster to resume I/O rather than following the rule set. VMware HA detects the failure of the virtual machines and restarts the virtual machines automatically at the surviving site with no external intervention. It is important to note that, a data


unavailability event can occur when there is not a full site outage but there is a VPLEX outage on cluster-1 and the virtual machine is currently running on the ESXi server attached to cluster-1. If this configuration also contains a VPLEX Witness, the witness recognizes the outage and recommends cluster-2 resume I/O rather than following the rule set. However, VMware vSphere does not recognize these types of failures and does not automatically move the failed virtual machines to the surviving site. To protect against this, one can leverage the cross-connectivity feature of the VPLEX storage system. Alternatively, users can intervene and manually move the virtual machine to the cluster-2 ESX server to provide access to the data. These options are not discussed in this paper, however, information on this can be found in papers in the References section.

VPLEX Failure Handling

There are numerous, excellent documents available that cover all the various failure scenarios that can impact a VPLEX environment. Therefore this document will not attempt to cover them but rather includes them in the References section so the customer may review them before deployment.

VMware Cluster Configuration with VPLEX Metro Witness

VMware cluster configuration

Figure 52 shows the recommended cluster configuration for VMware deployments that leverage devices presented through EMC VPLEX Metro with VPLEX Metro Witness employed. It can be seen from the figure that VMware vSphere consists of a single VMware cluster. The cluster includes two VMware ESXi hosts with one at each physical data center (referred to here as North Boston and South Boston). If site HA is also desired, two or more ESXi hosts could be configured at each site. Also shown in the figure, as an inset, are the settings for each cluster. The inset shows that VMware DRS and VMware HA are active in the cluster and that VM Monitoring is activated.


Figure 52. Configuration of VMware clusters with EMC VPLEX Metro utilizing VPLEX Metro Witness

To take full advantage of the HA cluster failover capability in a VPLEX Metro cluster that employs VPLEX Witness, it is necessary to create DRS Groups and then Rules that will govern how the VMs will be restarted in the event of a site failure. Setting these up is a fairly simple procedure in the vSphere Client. This will allow restarting of VMs in the event of a failure at either North Boston or South Boston.

VMware DRS Groups and Rules

To access the wizards to create DRS Groups and Rules, right-click on the cluster (Metro_Cluster) and navigate to VMware DRS. Leave the automation level at “Fully


automated” to permit VMware to move the virtual machines as necessary as in Figure 53.

Figure 53. Setting DRS to fully automated

For DRS groups, one should create two Virtual Machines DRS Groups and two Host DRS Groups. In Figure 54 four groups have been created. North_Boston_DRS_VM_Group contains those VMs associated with cluster-1 and North_Boston_DRS_Host_Group contains those hosts associated with cluster-1. The cluster-2 setup is similar.


Figure 54. Creation of DRS Groups for hosts and virtual machines

Hosts and VMs are added respectively to each group and this is seen in Figure 55.


Figure 55. Adding hosts and virtual machines to appropriate DRS groups

Now that the DRS groups are in place, rules need to be created to govern how the DRS groups should behave when there is a site failure. There are two rules, one that applies to cluster-1 and one that applies to cluster-2. The rule North_Boston_DRS_VM_Rule, seen in Figure 56, dictates that the VMs associated with cluster-1 (through the DRS group) “should run” on the hosts associated with cluster-1 (again through the DRS group) – vice-versa for cluster-2’s VMs. It is important that the condition for both rules is “should run” and not “must run” since this gives flexibility for the VMs to start-up on the two hosts that survive in a site failure. Each rule will permit the VMs associated with the failing cluster to be brought


up on the two hosts that are part of the site that did not fail, and most importantly, to automatically migrate back to their original hosts when the site failure is resolved.

Figure 56. Creation of rules for DRS groups


These configurations along with VPLEX Metro Witness will ensure that a site failure will not completely bring down the environment.

Best practices with VMware HA and VPLEX Metro

Follow these best practices when configuring a VPLEX Metro in a campus:

Configure the front end with a stretched layer-2 network so that when a virtual machine moves between sites, its IP address can stay the same.

Use DRS host affinity rules if DRS is enabled. Host affinity rules keep virtual machines running in the preferred site as long as the virtual machines can, and only moves the virtual machines to the non-preferred site if they cannot run in the preferred site.

Deploy the VPLEX Witness with this type of solution to avert some system-wide data unavailability events.

For more detail on HA and FT best practices see the white paper Using VMware Fault Tolerance and High Availability with VPLEX™ Metro HA for Ultimate Availability on Support.EMC.com.

Oracle Real Application Clusters (RAC) on VPLEX Metro

Extended RAC and Oracle Clusterware with VPLEX Witness

As previously defined, Extended Real Application Clusters has server nodes deployed in different physical locations, but maintains the strict latency requirements and disk sharing. Such a deployment is typically in a metro or campus environment. VPLEX Metro provides the perfect solution for Extended RAC and is fully supported by Oracle.

One of the benefits of VPLEX Metro is the avoidance of a third site for the Oracle voting disks which is usually a requirement of Extended RAC. A third site prevents a split-brain situation where the Oracle nodes are unaware there has been an interconnect issue and both sides continue to receive I/O. A VPLEX Metro does not need the third site because it uses VPLEX Witness in a third site fault domain as discussed previously in this paper. The behavior of Extended RAC with VPLEX Witness is as follows:

If there is an interconnect failure between Oracle RAC nodes that is unrelated to VPLEX, Oracle Clusterware will reconfigure based on majority rules and access to the voting disk.

If there is a VPLEX dual-wan partition failure or a site failure, VPLEX will initiate site preference rules with VPLEX Witness guidance and continue I/O at the designated site. The Oracle RAC node(s) accessing storage at the surviving VPLEX cluster site will still have access to the voting disks and therefore Oracle Clusterware will reconfigure the cluster in accordance with this.


For additional detail on using Oracle Extended RAC with VPLEX see the white paper, Oracle Real Application Clusters (RAC) on Extended Distance Clusters with EMC VPLEX™ Metro, on Support.EMC.com.

Oracle on VMware

Oracle deployment on virtual hardware should essentially be no different than physical hardware. In other words you can achieve all the scaling benefits of the hypervisor without being concerned that Oracle will run differently than when run on a physical host. Like its physical host counterpart virtual deployments should follow best practices for Oracle databases. Other recommendations mirror physical deployments and can be found at the VMware website. The References section includes some of these documents.

Oracle Real Application Clusters (RAC) and Oracle E-Business Suite

The Oracle E-Business Suite implementation (default demo database VIS) used in this paper installs as a single instance on a single node. If the user desires to use RAC, he must convert the database from single to multiple instances. For standalone databases, this is not a trivial task but for E-Business Suite environments it is even more complicated. Oracle provides a number of support notes to guide the user through the implementation. There are two components involved – the applications and the database. Because Oracle Applications is integrally tied to the database, it requires an upgrade just like the database. There are numerous patches and configuration changes that must be completed before the database can be moved to RAC. The Oracle notes are extensive and detailed and version specific for the applications and the database. They are also frequently updated with new information. It is therefore impossible to properly explain the process used in this environment beyond the high-level one already mentioned. The support notes used in this upgrade are included in the References for the user’s reference.

The focus here will be on one possible upgrade path to RAC in an Oracle E-Business Suite Environment. Oracle provides for the conversion to RAC on a number of different paths. The one used in this paper was a separate install of the 11.2.0.3 binaries (grid and database), a conversion to ASM, and an upgrade of the applications database both to the new database version and to Extended RAC. The paper will address the pertinent component which is the installation of RAC on VMware with VPLEX. It is this that will provide the foundation for both the RecoverPoint configuration and VMware SRM setup.

Oracle Real Application Clusters (RAC) on VMFS

RAC on VMware can be configured with either Raw Device Mappings (RDMs) or vmdks. For this solution vmdks were chosen to reduce complexity.

Though VMware SRM and the RecoverPoint SRA support replicating physical Raw Device Mappings (RDMs), VMFS was chosen foremost for its greater simplicity. Using RAC on VMware requires that certain steps are taken to enable the virtual machines running as RAC nodes to access the same vmdks. Oracle has a clustering technology,


Oracle Clusterware9, which contains the intelligence to ensure that two or more RAC nodes can access the same disk without causing data issues. While VMware will allow multiple VMs to access the same vmdk when used for such technologies as VMware Fault Tolerance (FT) where only one VM is writing, by default it does not permit multiple virtual machines to write to the same vmdk. If such a restriction were not in place, VMware could not guarantee any order of writes coming from the virtual machines which could very easily lead to data corruption and loss.

VMware provides the means to override this protection through the multi-writer flag that is set on a vmdk by vmdk basis. The process for this can be found in VMware KB article 1034165. VMware specifically calls out third-party clustering software like RAC for this type of change. One of the pre-requisites for setting the flag is that the Oracle vmdks designated for the database need to be created as type eagerzeroedthick. This is in line with VMware’s best practices for Oracle databases on VMware.

Oracle Clusterware deployment

Oracle Clusterware and Automatic Storage Management (ASM) are deployed on VPLEX distributed volumes and, as previously discussed, without the need for a third site for the voting disks. EMC recommends deploying the Clusterware files, OCR and voting, on their own ASM disk group using “Normal” redundancy, while using “External” redundancy for the other disk groups.

Oracle Extended RAC installation

The following are the basic steps undertaken when configuring Oracle RAC on VMware for this environment. These steps were taken on each of two nodes in the RAC environment. As this is being done on a VPLEX Metro environment, it is by default an Extended RAC configuration. The steps below assume that VPLEX distributed devices have been presented to both ESXi hosts in the HA cluster.

Note that in the environment in this paper two RAC nodes were used and that they were installed independently and with independent software homes. There are a number of options available when installing RAC nodes, particularly on VMware. Users may choose to install some components on a single node (e.g. OS, disks, etc.) and then create a template from it. Using that template they can create multiple RAC nodes. At that point they may choose to use a shared home for the Oracle binaries. These options are strictly preferences of the user doing the install and do not impact the functionality of the cluster, though they can make patching easier. Create VMFS on each of the distributed devices presented from the VPLEX. There will be some for the OS and binaries and others for the ASM setup for the Oracle RAC database.

Procedure

1. Configure an NTP client for use in the environment.

9 In 11g R2, the Oracle Clusterware and Automatic Storage Management (ASM) form the Oracle Grid Infrastructure.


2. Create a VM with Guest OS set to Oracle Linux 4/5/6 (64-bit). Create a single vmdk to hold the OS binaries, and another vmdk to hold the Oracle Application binaries.

3. Add a second NIC. Be sure both NICs are set to VMXNET3. These NICs should be on VMware networks configured on their own physical adapter.

4. Install Oracle Enterprise Linux, VMware tools and the ASMLIB components. Configure ASMLIB.

5. Configure NTP on the node.

6. Create vmdks for the Oracle database and CRS/voting disks. The configurations may vary. For this environment multiple disks (TDEVs) were created for the cluster files, data, redo, and archive. All vmdks need to be eagerzeroedthick (EZT), otherwise there will be issues such as an inability to restart the VM. Be sure the VMAX thin pools housing the disks behind the VPLEX distributed devices are not oversubscribed or the disk creation may fail. The disks should be assigned to a Paravirtual SCSI adapter.

7. Use VMware KB article 1034165 to set the multi-writer flag10 on the vmdks, allowing more than one node to access the disk.

8. Using fdisk configure a single partition on all the disks. It is important to align the partition on the disk since neither Oracle nor VMware will do this automatically. VMware ensures the VMFS is aligned but not the file systems on vmdks. VPLEX requires a 4 KB offset unless Symmetrix VMAX is backing the VPLEX. When this is the case 64 KB should be utilized as in this environment.

9. Use oracleasm to create the ASM disks on one of the hosts. Then use oracleasm to rescan on the second host and discover the ASM disks.

10. Install and configure Oracle Grid Infrastructure 11.2.0.3. The Clusterware install will use a single ASM disk group during installation. Set this disk group to “Normal” redundancy which will ensure multiple voting disks as Oracle automatically determines the number of files based on redundancy. This ASM disk group need not be large as the Clusterware files are small.

11. Create the remaining ASM disk groups using the ASM GUI or ASM CLI, being sure to set redundancy to “External”. Oracle and EMC both recommend using this redundancy due to the superior availability of not only VPLEX but the EMC Symmetrix VMAX which backs the VPLEX in this environment. The Clusterware disk group is the only one requiring a “Normal” redundancy for previously noted reasons.

12. Install Oracle database binaries on both nodes at once through the installer.

13. Convert the Oracle E-Business Suite Demo database to ASM and RAC.

10 Use of the multi-write flag renders Storage vMotion inoperable for the node.


Networking

The network configuration used in this environment was limited to two physical NICs on the ESXi hosts. A best practice would be to separate the management network, vMotion network, Oracle public and private networks. Figure 57 shows the configuration with just two NICs:

Figure 57. vSphere networking for Oracle RAC

After the Oracle Applications’ database is converted to RAC, the database administrator should be able to run a successful status check that should yield results similar to Figure 58. It displays the status of an Extended RAC node after installation and conversion.


Figure 58. Oracle RAC status on production node

To see detail of one of the VPLEX distributed devices, VSI provides a complete view in Figure 59.


Figure 59. VSI showing VPLEX distributed device anatomy

EMC RecoverPoint The EMC RecoverPoint product family provides a comprehensive data protection solution for enterprise and commercial customers, providing integrated continuous data protection and continuous remote replication to recover applications to any point in time.

RecoverPoint systems enable the reliable replication of data over any distance; within the same site (CDP), to another distant site (CRR), or both concurrently (CLR). Specifically, RecoverPoint systems support replication of data that applications are writing over Fibre Channel to local SAN-attached storage. The systems use existing Fibre Channel infrastructure to integrate seamlessly with existing host applications and data storage subsystems. For remote replication, the systems use existing IP connections to send the replicated data over a WAN, or use Fibre Channel


infrastructure to replicate data asynchronously or synchronously. The systems provide failover of operations to a secondary site in the event of a disaster at the primary site.

Here are some of the definitions of the components of RecoverPoint.

RPA – RecoverPoint Appliance. The hardware that manages all aspects of data protection. One RPA can manage multiple storage groups, each with differing policies.

A minimum of two and a maximum of eight RPAs are installed at each site, located in the same facility as the host and storage. The set of RPAs installed at each site is referred to as an “RPA cluster”. If one RPA in a cluster fails, the functions provided by the failed RPA are automatically moved to one or more of the remaining RPAs.

The RPAs at the production site transfer the split I/O to the replica site.

The RPAs at the replica site distribute the data to the replica storage.

In the event of failover, these roles can be reversed. The same RPA can serve as the production RPA for one consistency group and the replica RPA for another.

Volumes – All RecoverPoint volumes can be hosted on VPLEX. In practice, some volumes may be hosted on VPLEX and other hosted on non-VPLEX storage. For example, the repository volume for an existing RPA cluster cannot be moved. If you are installing VPLEX into an existing RecoverPoint configuration, the repository volume is already configured on non-VPLEX storage.

The following types of volumes are required in all RecoverPoint configurations:

Repository volume – A volume dedicated to RecoverPoint for each RPA cluster. The repository volume serves all RPAs of the particular RPA cluster and the splitter associated with that cluster. The repository volume stores configuration information about the RPAs and RecoverPoint consistency groups.

There is one repository volume per RPA cluster.

Production volumes – Volumes that are written to by the host applications. Writes to production volumes are split such that they are sent to both the normally designated volumes and RPAs simultaneously. Each production volume must be exactly the same size as the replica volume to which it replicates.

Replica volumes – Volumes to which production volumes replicate. The replica volume must be exactly the same size as its production volume.

Journal volumes – Volumes that contain data waiting to be distributed to target replica volumes and copies of the data previously distributed to the target volumes. Journal volumes allow convenient rollback to any point in time, enabling instantaneous recovery for application environments.

Snapshot/PIT – A point-in-time copy that preserves the state of data at an instant in time, by storing only those blocks that are different from an already existing full copy of the data.


Snapshots are also referred to as Point-in-Time (PIT). Snapshots stored at a replica journal represent the data that has changed on the production storage since the closing of the previous snapshot.

Logged (physical) access – Used for production recovery, failover, testing, and cloning a replica. Also referred to as image access in this white paper.

Direct access – This access mode can only be enabled after logged access, or virtual access with roll, are enabled. Used for extensive processing with a high write-rate, when image access is needed for a long period of time (and may not have the journal space to support all of the data written to the image access log in this time), and when it is not required to save the history in the replica journal (the replica journal is lost after direct access).

RecoverPoint 4.0

RecoverPoint released its 4.0 version of the software in Q2 2013. This version has many enhancements including:

RecoverPoint/VE readiness

Multi-site

vRPA with VNX splitter

Scale and Performance

New GUI over EFX – Unisphere for RecoverPoint

Customer upgradeable

Electronic licensing

RecoverPoint CDP and SRDF

With the release of RecoverPoint 4.0, Enginuity Q2 2013 SR and Solutions Enabler V7.6, RecoverPoint CDP and SRDF can now co-exist on the same source devices. This provides the ability to have a local copy of the data to protect against logical corruption while replicating to a remote site for disaster recovery.

VPLEX Metro RecoverPoint Configuration

VPLEX Metro and RecoverPoint CRR/CLR

In VPLEX Metro/RecoverPoint CRR/CLR configurations, I/O is:

Written to both VPLEX clusters (normal behavior)

Split on one VPLEX cluster to replica volumes located both at the cluster and at a remote site.

RPAs are deployed at one VPLEX cluster and at a third site.


The third site can be an independent VPLEX cluster. Figure 60 is a diagram of this deployment model, the same that is used in the environment detailed in this white paper.

Figure 60. High-level environment diagram of VPLEX and 3rd site with RecoverPoint

Although the RecoverPoint splitter is resident in all VPLEX clusters, only one cluster in a VPLEX Metro can have RPAs deployed. This configuration supports unlimited points in time, (with granularity up to a single write) for local and distributed VPLEX virtual volumes.

RecoverPoint appliances can be deployed at only one VPLEX cluster in a Metro configuration.

All RecoverPoint-protected volumes must be on the preferred cluster, as designated by VPLEX consistency group-level detach rules.

Customers can recover from operational disasters by quickly returning to any PIT on the VPLEX cluster where the RPAs are deployed or at the third site.

Application event aware based rollback is supported on VPLEX Metro distributed/local virtual volumes for Microsoft SQL, Microsoft Exchange and Oracle database applications.

If the VPLEX cluster fails, customers can recover to any point in time at the remote (third) site. Recovery at remote site to any point in time can be


achieved through VMware vCenter Site Recovery Manager as detailed in this white paper.

This configuration can simulate a disaster at the VPLEX cluster to test RecoverPoint disaster recovery features at the remote site.

Unisphere for RecoverPoint Unisphere for RecoverPoint is the new GUI interface in version 4.0 for managing the RecoverPoint environment. It provides easy to use wizards to complete all tasks related to configuring and managing a RecoverPoint environment. Figure 61 presents the dashboard of Unisphere which includes three tabs:

Overall Health

System Monitoring

System Events

In the Overall Health tab there are four sub-panels containing:

The RPA Cluster Map

System Traffic

Alerts

Consistency Group Transfer Status


Figure 61. Unisphere for RecoverPoint dashboard

In addition to the dashboard statistics, greater performance detail can be obtained in the Statistics tab for each consistency group. An example of this is in Figure 62.


Figure 62. Unisphere for RecoverPoint statistics view

RecoverPoint Configuration for the Oracle Environment Once the RecoverPoint hardware and software has been successfully installed, it must be licensed for use with CDP, CRR, or both at the local and remote site. The “Getting Started Wizard” in Unisphere for RecoverPoint will guide the user through the installation of the licenses required to use the product. Due to the many uses of this RecoverPoint environment, both CRR and CDP licenses are required at both the local and remote site.

Licensing RecoverPoint

The wizard is a step-by-step process to add the licenses. Both CRR and CDP can be licensed on the same site through the use of two license files at each site. The introduction screen is seen in Figure 63.


Figure 63. RecoverPoint Licensing wizard

Select “Add” then supply the appropriate file which contains the license information as in Figure 64.

Figure 64. Add license to RecoverPoint

Select “OK” then “Next Enable Support” which brings up Figure 65. Fill in the appropriate information for the environment and select “Finish”.


Figure 65. Enable support in RecoverPoint license wizard

Follow these same steps to add the remaining license files. When complete it should appear as in Figure 66.


Figure 66. RecoverPoint CRR and CDP licenses for local and remote sites

VPLEX credentials on splitter

Once configured and licensed, you may have a warning concerning the VPLEX splitter, seen in Figure 67.

Figure 67. RecoverPoint warning to enter credentials for VPLEX


RecoverPoint requires the VPLEX credentials to be entered. These should be for the management server of the VPLEX cluster to which the RecoverPoint appliance is associated, in this case cluster-2. An example is seen in Figure 68.

Figure 68. Entering credentials for VPLEX Metro

RecoverPoint Configuration

RecoverPoint Journals

RecoverPoint journals are highly active volumes in a RecoverPoint configurations. In general, the journals on the target array will experience 3-5 times the I/O as the source. The reason for this is simply the nature of how RecoverPoint works. A highly active thin device should be backed by a highly performing disk technology. In this configuration FC RAID-1 was chosen for the journals. This will ensure the proper performance. The use of FAST VP, however, will provide the benefit of moving non-active data to SATA RAID-6, or very highly active data to EFD RAID-5. For the device configuration EMC recommends using striped metavolumes, either 4 or 8-way. The size of the volumes should be equal, and it is best to use the same number of volumes as there are FA ports available to RecoverPoint. For this configuration,


therefore, 4, 4-way striped metavolumes were used for each RecoverPoint consistency group. RecoverPoint will stripe across the journal volumes, distributing the workload among the FAs.

RecoverPoint Replicas

The replicas on the 3rd site receive 2 times the amount of I/O as the production volumes due to RecoverPoint activity. It is acceptable to place the replicas on slower performing disks than production, however at failover those replicas become production volumes. If FAST VP is in use, as is recommended, then the volumes are likely to be on the correct tier already.

Consistency Group Creation

This will follow through the process of creating a consistency group in the RecoverPoint environment. From the Unisphere for RecoverPoint dashboard select the menu “Protection” and then “Protect Volumes” seen in Figure 69 which will start the wizard.

Figure 69. Consistency Group creation in Unisphere for RecoverPoint

The first screen defines the consistency group name along with the volumes to be replicated. Use the checkboxes to select which volumes. If the volumes were named according to function then this is fairly straightforward. Type in a group name and a production name. In this example the production name is prefixed with “Local_” to distinguish it from the remote site which will be defined in a subsequent screen. The RPA Cluster will default to the local one if Unisphere is being accessed from the local site. Details are in Figure 70.


Figure 70. Select production volumes for local site

Next the production volumes need to be assigned in Figure 71. Once again this is a selection of volumes through checkbox. The volumes are easy to identify as the naming convention designates which volumes function as what component of the database. Each consistency group will be assigned four journal volumes which are configured to be 20% of the total amount of space being replicated – the best practice for journal volumes. As the storage backing the environment is VMAX, the journal volumes are striped metavolumes which will provide the best performance, also previously noted as a best practice.


Figure 71. Define production journals for local site

Now the remote site needs to be defined. A Copy Name is supplied, and in Figure 72 one sees the prefix “Remote_” is used. The RPA Cluster should default to the remote site. The production volumes need to be associated with a similarly sized volume on the remote site.11 The sub-selection pairing screen for each device is shown in Figure 73. These remote volumes are the ones configured on the third site VMAX array. The Replication Mode is asynchronous as we are traversing Boston to San Francisco for a true DR scenario.

11 There is the capability of using a larger sized remote volume.


Figure 72. Map production volumes to remote 3rd site volumes

Figure 73. Sub-selection screen for mapping devices

Following the pairing, the remote site needs journal volumes also. This select is seen in Figure 74 and the volumes follow the same best practices as the local journals.


Figure 74. Define copy journal for remote site

At the final screen in Figure 75 there is the ability to modify the group if something was done incorrectly. Prior to this screen the ability to backtrack was always available.


Figure 75. Consistency Group Creation Summary

Setting external management for a consistency group

Figure 76 shows how to set the consistency group in RecoverPoint to be managed by an external application, in this case VMware vCenter Site Recovery Manager, or SRM. It is critical that this is set before attempting to use the RecoverPoint SRA in VMware SRM. Failure to make this change will result in an inability to see any paired devices between the VPLEX and RecoverPoint.


Figure 76. Set VMware SRM as the external management for the consistency groups

Group Set creation

We will also take this opportunity to create a Group Set in Figure 77 which will bookmark all our consistency groups at the same time with a single tag. This done through the VPLEX GUI -> Protection -> Manage Group Sets.


Figure 77. Create a group set for bookmarks

This is not the only type of group set that might be used in a RecoverPoint environment. It is used in this case as the implementation is for VMware SRM. Note however that the setup is still such that the Oracle database has been separated into consistency groups that would allow the Oracle database data to be image accessed at a different point-in-time than the archive logs – allowing a hot backup scenario. In customer environments this type of granularity may be desired so that if the environment were used not only in VMware SRM but also for other purposes such as backup or development/test, the different consistency groups could each have a different bookmark. In a dual-purpose type VPLEX/RecoverPoint/SRM environments remember that external management by SRM must be disabled before a manual image access is attempted otherwise an error will be generated as in Figure 78.


Figure 78. Cannot use GUI for image manipulation while SRM owns management

VMware SRM Installation Version 5.1 of VMware SRM was used in the configuration. The installation of SRM will not be covered herein. Please see the appropriate VMware documentation for installation instructions. The assumption going forward is that VMware SRM is successfully installed and configured.

RecoverPoint SRA

The RecoverPoint SRA is required in order to use VMware SRM with RecoverPoint. VMware SRM interacts with the SRA to perform the necessary RecoverPoint activities that permits image access. The installation of the SRA requires a couple pre-requisites for installation. One is that SRM is installed and the second is that Java Runtime Environment (JRE) Version 6 Update 7 is present on the VMware SRM host.


Note however that if the RecoverPoint SRA does not find the JRE it will install a pre-canned version. Unfortunately the JRE that RecoverPoint installs will only allow the SRA install to complete successfully, but the adapter will not operate properly. If the SRA is not working properly, examine the log files for information as to the failures, one of which may be a JRE problem.

We will step through the installation of the RecoverPoint SRA adapter. The installation requires almost no user interaction and can be stepped through very easily. In the final screen, Figure 79, the installer warns that the VMware SRM service may need to be restarted. If the SRA adapter is not recognized in VMware SRM that could indicate a restart is necessary. Note that doing so will disrupt any running SRM processes.

Figure 79. Completion of RecoverPoint SRA installation

VMware SRM Protection and Recovery Groups As all the pieces are now in place, we can create the VMware SRM Protection Groups and Recovery Plans to conduct failover testing. The array manager is added first, both to the protected site and the recovery site. This is completed through the VMware SRM configuration screens and the VMware SRM select screen in Figure 80.


Add a RecoverPoint Array Manager

Figure 80. SRM select screen

In Figure 81, after selecting the appropriate site in the select screen, the user chooses the “Add Array Manager” and provides the detail from the RecoverPoint environment. The steps are outlined below where the IP address of the management server on the local site, 10.108.246.150, is supplied.


Figure 81. Adding an array manager to VMware SRM

After the array managers are added to each site, enable the pairs. Though each site needs an array manager configured, enabling the pairs only needs to be completed on one site. Enable the manager by selecting the “Enable” blue word under the Actions column in Figure 82.

Figure 82. Enabling the array manager in VMware SRM


Successful enabling will result in the pairs displaying at each site as seen in Figure 83.

Figure 83. Enabled device pairs in SRM

If the RecoverPoint consistency group is not set to external management by SRM, the pairs for that group will not appear in this display. Return to RecoverPoint, enable external management by SRM as shown in Figure 76, and run a Refresh from this screen.

VMware SRM Protection Group creation

To create a protection group, use the menu from Figure 80 and select Protection Groups and then select Create Protection Group seen in Figure 84.


Figure 84. Create VMware SRM Protection Group

Next follow the wizard detailed in Figure 93, supplying the requested information. Note that at Step 2 a single datastore group is selected. For this configuration the goal is to create a crash-consistent copy of the Oracle environment. While it is possible to involve a hot-backup mode of the Oracle database, it would require a different procedure and additional steps, including manual copying of log files over to the remote site. A crash-consistent Oracle database, however, can still be used for backup and recovery.


Figure 85. Creating a protection group in SRM

The configured virtual machines are included here in Figure 86:

Figure 86. Configured virtual machines in protection group


When the protection group is finished, create the recovery group. Note that both the protection groups and recovery plans can be created while logged in to either the local or remote VMware SRM site. The five steps are present in Figure 87.

Figure 87. Creating a recovery group in VMware SRM

The recovery plan is set to automatically start the virtual machines in the Oracle environment. A few things of note about the recovery plan are important to emphasize:

All four virtual machines are being started on the same ESXi host. This was the limitation of this environment. The host in question has enough resources to run


all four machines, but in a customer environment it is expected that an HA cluster would exist at the third site. That being the case, the third site should be configured in a manner similar to the production one in terms of VMware DRS. See VMware Cluster Configuration with VPLEX Metro Witness for detail.

The virtual machines in the recovery plan are set to startup automatically. If the Oracle RAC database will be used for backup or need to be rolled forward, then the plan should be changed to not have the virtual machines startup. For details on how to use a crash-consistent Oracle database for backup or recovery please see the white paper on EMC.com, Using EMC SRDF/TimeFinder and Oracle database 10g/11g. Although the paper covers the EMC technologies SRDF and TimeFinder, the focus is on crash-consistent databases which EMC RecoverPoint produces in this environment.

In an Oracle Applications environment, the database must be accessible before the application services startup. The recovery plan can be customized to allow different priorities when starting up the virtual machines as well as scripts to be executed. There are numerous ways to configure the environment to automate the Oracle Applications coming up in a test or DR scenario.

VMware SRM Test Failover The following sections detail running the previously created VMware SRM recovery plan. It utilizes the VSI feature RecoverPoint Management version 5.5.

VSI: RecoverPoint Management The RecoverPoint Management 5.5 feature is an enhancement to VSI that allows the user to select Point-in-Time (PiT) copies for use in VMware SRM failover testing or disaster recovery scenarios when used in conjunction with RecoverPoint 4.0 P1. The RecoverPoint Management (RM) 5.5 feature is accessed and run from within VSI. It is not intended to replace the RecoverPoint GUI console in any way as it is only used in conjunction with Site Recovery Manager. In fact, the use of the feature is only possible if RecoverPoint is previously configured to allow VMware SRM to manage the environment for the purpose of image access.

In order to use VSI RM with the VMware SRM environment it is necessary to provide credentials for both the VMware SRM servers and the RecoverPoint server. This step is detailed in Figure 88.


Figure 88. Adding credentials for SRM and RecoverPoint in VSI


With the entries in place seen in Figure 89, VSI will be able to discover information about the RecoverPoint environment, such as consistency groups and bookmarks, which will permit the setting of a point-in-time (PiT) for a consistency group.

Figure 89. Registered RecoverPoint and SRM Servers in VSI

The integration of RecoverPoint and VMware SRM does not require VSI RM to function properly. VSI RM provides the ability to set a PiT rather than use the last successful bookmark when testing the failover from production. If VSI RM is not present in the environment, VMware SRM through the RecoverPoint SRA will use a real-time, crash-consistent image across all RecoverPoint consistency groups in the protection group.

RecoverPoint Failover Testing with VSI The following section will detail a SRM failover test using the VSI RM feature and PiT selection.

In previous sections, examples have been provided for creating consistency groups in VPLEX and then creating consistency groups in RecoverPoint. These groups mirror each other as a best practice. In any case a VPLEX consistency group is required for any devices that will be replicated with RecoverPoint.

Point-in-Time assignment in VSI for RecoverPoint Consistency Groups

When executing a recovery plan for VMware SRM with RecoverPoint, by default the latest consistent image across the consistency groups in the protection group will be used. For many customers this might be acceptable, but for others they will want the capability that the VSI RecoverPoint Management feature provides which is PiT selection. With VSI RM a PiT image can be applied, such as the one being created every ten minutes by the Group Set previously outlined in Group Set creation. Setting the PiT is done on the production (protected) site at the VM level. Figure 90. VSI RecoverPoint Management interface shows the interface along with an active VMware SRM failover test using the latest consistent image.


Figure 90. VSI RecoverPoint Management interface

In Figure 91 is the detail how a PiT is applied instead of using the current image. It is only necessary to apply the PiT to one entry for each consistency group. Note that you must apply a PiT to every consistency group involved in the protection group.


Figure 91. Applying a PiT in VSI for RecoverPoint consistency groups

There is an easy way to check that this has been done correctly. In VSI there is a secondary view called “Protection Groups” where all the consistency groups related to the group are shown. The view in Figure 92 is shown after all PiTs have been applied.


Figure 92. Assigned PiTs as displayed in VSI

The user can also set an “Expiry Time” on the image so that it will not be used after a certain date. Start by selecting a consistency group with a PiT set. Choose the “Set Expiry Time” blue wording to pop-up the dialog box. Select a date as in Figure 93 to expire the PiT and choose OK. The Expiry Time column will be updated with the date. It is recommended that the expire time be the same for all consistency groups involved in the SRM testing. If no expire time is applied, the PiT will be used any time a SRM test failover is conducted assuming the PiT is not manually changed.


Figure 93. Setting an expire time for a RecoverPoint PiT in VSI

Test Failover

To start a failover test, on the remote vCenter navigate to the Site Recovery icon from the home page of vCenter see in Figure 94.

Figure 94. Site Recovery Manager icon in vSphere vCenter Server

Login to the site then select the EBS_Recovery_Group plan previously created. At this point the test can be run. In the screen below, Figure 95, it demonstrates the three steps to execute. First select “Test” from the options in the panel. When the second dialog box comes up, uncheck the box to replicate recent changes. This box is irrelevant to the RecoverPoint SRA but if it is left checked then SRM does some extra


testing adding time to the failover test. Select “Next” then “Start” to begin the failover test.

Figure 95. Testing the EBS environment failover in SRM with RecoverPoint

While the test is running, the status of the RecoverPoint consistency groups on the remote site can be watched to see when image access is enabled. Figure 96 shows the transformation from one state to the other.


Figure 96. RecoverPoint image access during SRM test failover

With the enabled images, the virtual machines are brought up into the internal environment for testing. Figure 97 shows the step in the recovery where the virtual machines are in the midst of starting.


Figure 97. SRM recovery steps, virtual machine power-on

Shortly after the power-on of the virtual machines, the test finishes successfully as shown in Figure 98.

Figure 98. SRM successful failover test


After completion of the failover test, we can see that all the datastores are attached, by default resignatured and renamed, in Figure 99.

The “snap-xxxx” prefix seen in Figure 99 can be turned off through VMware SRM so that the datastore names are the same as production. This might be desirable if there are customer scripts that rely upon the datastore names.

Figure 99. Viewing datastores for RAC node at failover test site

Now attach to the console of one of the RAC nodes, dsib2124, which is running on the internal network (no IP conflict) and run the same commands as we did in Figure 58 (Figure 100 shown).


Figure 100. RAC node status at failover test site

The test is successful with all VMs up and running and the Oracle Applications environment ready for access.12

RecoverPoint PiT with VSI in Disaster Recovery

The functionality detailed in this section is only enabled with RecoverPoint 4.0 P1 and higher.

As demonstrated in the previous section on SRM testing with RecoverPoint, in order to obtain PiT bookmarks of the RecoverPoint consistency groups, VSI queries the RecoverPoint appliance from the production (protected) site. In a real disaster, however, the production site is unavailable and VSI RM is unable to execute the query

12 Oracle Applications would require some customization to the Application Tiers to automate the process of the services coming up.


to obtain the PiTs. If an SRM failover is executed at that point, as in Figure 101, RecoverPoint will use the last consistent copy (bookmark) across the consistency groups.

Figure 101. SRM Recovery - Planned Migration or Disaster Recovery

For many companies this will be perfectly acceptable because they will wish to have the disaster site as close as possible to the production site prior to failover. Some companies, however, may desire to recover to a particular point-in-time either to avoid logical corruption issues, or for other business reasons. If there is a requirement to use a PiT before the last consistent copy, there is a process that needs to be followed to enable PiT selection on the recovery site with VSI RM. It is outlined in the next section.

Enabling PiT with VSI RM on the Recovery Site

Once the decision is made to initiate a disaster recovery with a PiT, rather than the last consistent copy, the following steps need to be taken:

1. Begin by enabling an option within VSI RM that permits a test failover with the RecoverPoint environment in a split state. To set this, bring up the vCenter


recovery site environment. Navigate to the EMC icon in the home page, previously seen in Figure 4, which will bring up the VSI management screen. Once in VSI, take the following steps which are graphically represented in Figure 102:

1. Select the RecoverPoint Management feature from the left-hand menu.

2. Highlight the configured RecoverPoint Server. The configuration of these servers was demonstrated previously in Figure 88. This RecoverPoint Server should be on the failover site.

3. Choose the “AllowTestFailover” button at the top. 4. Confirm the selection. A task will appear for this action.

Figure 102. Allow test failover in VSI when protected site is unavailable

2. When the VMware task indicates a status of “Completed”, run a test failover with VMware SRM. As the production site VSI functionality will be unavailable, the failover copy will be either the last consistent copy OR a PiT that was previously set and whose expiration date (if any) has not passed. In order to


save time and reduce the total down-time of the disaster event, it may be useful to change the power-on state of the virtual machines to not power-on.

Note that if there was an unexpired PiT previously set and it is no longer available in the journals, the test failover will fail and the only option will be to perform the disaster recovery using the last consistent copy. If PiT recovery is critical for the environment in the event of a disaster, be sure to either clear the PiT in VSI after each test, or set an expiration date so that there is no chance the PiT could be purged from the journal before it expires.

3. Once the test failover is complete (prior to the Cleanup), all the PiTs will be available from the EMC VSI tab on the recovery site. Figure 103 demonstrates that the recovery site in San Francisco now has access to all the bookmarks (snapshots) that were available on the protected site before the disaster.

Figure 103. PiT select on the recovery site post-test failover

4. As was shown in Figure 91, apply the desired PiT to each of the consistency groups.13 For this example, in Figure 104, the desire is to have a PiT that is about 20 minutes prior to the disaster to avoid a logical corruption that

13 Unlike Figure 92 the Protection Groups view is unavailable on the recovery site.


occurred after that time. This is EBS_Group_Set 399332 and is seen in Figure 104.

Figure 104. PiT assignment on the recovery site before DR failover

5. Once the PiT is set, a Cleanup is required for SRM so that the real PiT can be applied during the disaster recovery failover depicted in Figure 101. The Cleanup is demonstrated below in Figure 105.


Figure 105. Running a "Cleanup" in VMware SRM after PiT selection

6. After the Cleanup is complete, if configured previously, modify the VMs that are part of the recovery plan so they power-on during the disaster recovery failover.

7. Run the final disaster recovery failover as in Figure 106. If it is a true disaster recovery scenario where the production site is unavailable, this task will return with warnings and errors, shown in Figure 107, despite bringing up the environment successfully. This is expected.


Figure 106. VMware SRM disaster recovery execution

Figure 107. VMware SRM expected errors in a true failover

When complete, RecoverPoint will show the remote site as being active, directly accessing the storage and recovered to the chosen PiT, while the local site is


unknown. The link will also either show an error or N/A as in Figure 108. The disaster recovery is now complete.

Figure 108. Remote RecoverPoint site active after DR failover

An important consideration when using SRM is that the time to perform a test or real failover is determined in large part by the PiT bookmark on RecoverPoint. The older the bookmark, the longer it will take RecoverPoint to enable image access, and consequently complete the failover. VMware SRM 5.x enforces a timeout of 300 seconds (5 minutes) for all Storage Replication Adapter (SRA) operations. If a SRA does not complete an operation within this time limit, the SRM will terminate the operation, possibly causing the environment to be in an undesired state. This timeout is more likely to be breached if the RecoverPoint PiT is not recent. For information on how to change the timeout value, refer to the relevant VMware SRM documentation.


Recommendations when using RecoverPoint with VMware SRM The following are general recommendations that should be followed when using RecoverPoint with SRM:

Group all SRM protection groups LUNs into a single consistency group or a number of RecoverPoint consistency groups. This grouping is to ensure that SRM protection groups LUNs are included in RecoverPoint consistency groups.

Ensure RecoverPoint consistency groups containing SRM protection group LUNs do not contain non-SRM protection group LUNs. Otherwise, if such LUNs are added to the consistency group, they will be handled by SRM.

Ensure RecoverPoint groups do not contain LUNs from more than one SRM protection group. Otherwise, SRM may attempt to operate concurrently on the same consistency group. This attempt may cause SRM operations to fail, or not behave as intended.

In addition to VMware SRM, you can use the RecoverPoint GUI and CLI to monitor the RecoverPoint system. However, it is not recommended to perform operations concurrently on the same consistency group using VMware SRM and RecoverPoint (GUI or CLI), as this may cause operations to interfere with one another. In general, however, RecoverPoint will prevent the user from doing this.

Conclusion EMC VPLEX Metro running the GeoSynchrony operating system is an enterprise-class technology that dissolves distance by distributing heterogeneous block storage devices over distance enhancing availability and mobility. EMC RecoverPoint brings continuous data protection and remote replication for on-demand protection and recovery at any point in time. RecoverPoint ensures this continuous remote replication without impacting performance. Using VPLEX Metro and RecoverPoint in conjunction with VMware and Oracle availability technologies, VMware SRM and RAC, provides new levels of availability and disaster recovery suitable for the most mission critical environments without compromise that go beyond any other solution on the market today.

These technologies provide the basis by which a customer can ensure high availability at both the hardware and software level – through the nature of VPLEX Metro and Oracle Extended RAC. Not only is the environment highly available, but it is protected with a robust disaster recovery solution, again through hardware and software. RecoverPoint in combination with VMware SRM match the benefits of an enterprise hardware disaster recovery solution with the benefits of an enterprise software solution. Managing all these technologies with easy to use GUI interfaces makes the job of administering it as seamless as the technology itself.


References

EMC Using EMC Symmetrix Storage in VMware vSphere Environments TechBook

http://www.emc.com/collateral/hardware/solution-overview/h2529-vmware-esx-svr-w-symmetrix-wp-ldv.pdf

Implementing EMC Symmetrix Virtual Provisioning with VMware vSphere

http://www.emc.com/collateral/hardware/white-papers/h6813-implting-symmetrix-vrtl-prvsning-vsphere-wp.pdf

Implementing VMware vStorage API for Storage Awareness with Symmetrix Storage Arrays

http://www.emc.com/collateral/software/white-papers/h10630-vmware-vasa-symmetrix-wp.pdf

The following are available on Support.EMC.com website:

Oracle Databases on EMC Symmetrix Storage Systems TechBook

Implementing Fully Automated Storage Tiering with Virtual Pools (FAST VP) for EMC Symmetrix VMAX Series Arrays

VSI for VMware vSphere Storage Viewer Product Guide 5.5 Product Guide

Unisphere for VMAX Product Guide

Unisphere for VPLEX Product Guide

EMC VPLEX with GeoSynchrony 5.1 Documentation

EMC VPLEX Metro 5.0 Witness Technology and Absolute High Availability

RecoverPoint 4.0 Documentation

VMware vSphere Storage ESXi 5.1

http://pubs.vmware.com/vsphere-51/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-51-storage-guide.pdf

vSphere Installation and Setup Guide

http://pubs.vmware.com/vsphere-51/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-51-installation-setup-guide.pdf

Virtual Machine Administration Guide

http://pubs.vmware.com/vsphere-51/topic/com.vmware.ICbase/PDF/vsphere-esxi-vcenter-server-51-virtual-machine-admin-guide.pdf

The following are available at the VMware.com website:

Oracle Databases on VMware Best Practices Guide














VMware vSphere Metro Storage Cluster Case Study

Oracle

The following are available at Support.Oracle.com website:

Oracle Applications Installation Guide: Using Rapid Install Release 12

Oracle Applications Concepts Release 12

Oracle Database 11g R2 Upgrade Guide

Using Oracle 11g Release 2 Real Application Clusters with Oracle E-Business Suite Release 12 [ID 823587.1]

Database Initialization Parameters for Oracle E-Business Suite Release 12 [ID 396009.1]

Database Documentation Resources for EBS Release 11i and R12 [ID 1072409.1]

using vplex metro and recoverpoint with vmware srm in an oracle extended rac e ... ·...

Documents