disaster recovery of oracle fusion middleware and oracle ... · pdf filedisaster recovery of...

Disaster Recovery of Oracle Fusion Middleware and Oracle Database Server

with EMC RecoverPoint Applied Technology

Abstract

Oracle database and application administrators face many challenges to managing the application and storage resources necessary for Oracle operations. This white paper outlines how EMC® RecoverPoint provides cost-effective local and remote replication of Oracle Fusion Middleware and Oracle Database Server as part of a disaster recovery solution.

August 2009

Copyright © 2009 EMC Corporation. All rights reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an applicable software license.

For the most up-to-date listing of EMC product names, see EMC Corporation Trademarks on EMC.com

All other trademarks used herein are the property of their respective owners.

Part Number h6450

Disaster Recovery of Oracle Fusion Middleware and Oracle Database Server with EMC RecoverPoint Applied Technology 2

Table of Contents Executive summary ............................................................................................5

Background.................................................................................................................................. 5 Introduction.........................................................................................................5

Audience ...................................................................................................................................... 6 RecoverPoint.......................................................................................................6

Advantages of RecoverPoint ....................................................................................................... 6 Local and remote recovery .......................................................................................................... 6 Replication modes ....................................................................................................................... 7 Oracle database protection with RecoverPoint............................................................................ 7 Federated environments and consistency groups....................................................................... 8

Use case objectives..........................................................................................10 Oracle/EMC environment block diagram ........................................................10 RecoverPoint validation architecture..............................................................12 CLARiiON array-side preparations..................................................................14 System configuration and storage ..................................................................15

Hardware configuration.............................................................................................................. 15 Software configuration ............................................................................................................... 16 Oracle Fusion Middleware, web tier, and Database volume layout........................................... 17 RecoverPoint CRR consistency groups..................................................................................... 18 RecoverPoint group sets and parallel bookmarks ..................................................................... 18

OCFS2 configuration ........................................................................................19 High availability for persistent stores ......................................................................................... 19 Use high-availability storage for state data................................................................................ 19

Network configuration......................................................................................20 Planning for disasters and planned downtime...............................................20

Initiate the replication process ................................................................................................... 20 RecoverPoint software installation and configuration ............................................................ 21 Starting replication.................................................................................................................. 21

Switchover procedures .............................................................................................................. 22 Switchback procedures.............................................................................................................. 24 Failover procedures ................................................................................................................... 26 Failback procedures................................................................................................................... 27

General recommendations...............................................................................28 Setting snapshots or manual bookmarks based on requirements............................................. 28 Periodic DR testing .................................................................................................................... 28 Event notification........................................................................................................................ 29 General I/O and sizing ............................................................................................................... 29

Conclusion ........................................................................................................29 References and resources ...............................................................................30

Oracle......................................................................................................................................... 30


EMC ........................................................................................................................................... 30 Appendix ...........................................................................................................31

Oracle DR terminology............................................................................................................... 31 RecoverPoint terminology.......................................................................................................... 32 RecoverPoint write splitters ....................................................................................................... 33 RecoverPoint image access modes .......................................................................................... 34

Virtual access (instant) ........................................................................................................... 34 Virtual access (instant) with Roll image in background.......................................................... 34 Logged access (physical)....................................................................................................... 35 Disable Image Access............................................................................................................ 35


Executive summary Oracle’s family of middleware products is comprehensive, standards-based application infrastructure software – from the leading Java application server to SOA and Enterprise 2.0 portals. Pre-integration with Oracle Applications, Database, and Enterprise Manager speeds implementation and lowers costs.

EMC® RecoverPoint is an important advancement in the area of data replication. RecoverPoint provides local and remote replication for heterogeneous servers and storage and enables multiple applications to have replication consistency with fine-grained control over local and remote recovery.

This white paper describes an application topology supporting core services provided by Oracle Fusion Middleware and Oracle Database Server. The data replication and disaster recovery products represented can address your specific application topology as it expands beyond the core represented here. This disaster recovery validation test incorporates the use of Oracle Application Server SOA Suite and Oracle Database Server 10gR2.

Oracle SOA Suite is a comprehensive, hot-pluggable software suite for the building, deployment, and management of a service-oriented architecture. This includes service-oriented application development, service-oriented applications and IT systems integration, and service-oriented management of business processes. It plugs in to a heterogeneous IT infrastructure and enables enterprises to adopt SOA incrementally.

These applications rely on hundreds of gigabytes or even terabytes of data and they share one common factor. They need a well-designed recovery plan in case of disaster. Oracle and EMC deliver key solutions that address the protection of mission-critical applications.

Enterprise deployments need protection from unforeseen disasters and natural calamities. One protection solution involves setting up a disaster recovery (DR) site at a geographically different location from the production site. The DR site may have equal or fewer services and resources compared to the production site. Application data, metadata, configuration data, and security data are replicated to the DR site on a periodic basis. The DR site is normally in a passive mode; it is started when the production site is not available. This deployment model is sometimes referred to as an active/passive model. This model is normally adopted when the two sites are connected over a WAN and network latency does not allow clustering across the two sites.

Background This white paper describes the setup and testing environment deployed to validate EMC RecoverPoint 3.1 with both Oracle Fusion Middleware and an Oracle database. This test configuration validates that the entire environment can be restored at a secondary site (DR site) in the event of a major production site failure. This test validates third-party replication using EMC RecoverPoint and Oracle Database Server as a joint solution that protects both database and non-database artifacts.

The white paper Disaster Recovery of Oracle Fusion Middleware with EMC RecoverPoint outlines an alternative solution approach where Oracle Data Guard is used to protect Oracle database files.

Introduction This white paper is a follow-on paper to Disaster Recovery of Oracle Fusion Middleware with EMC RecoverPoint. It addresses and explains the benefits of using EMC RecoverPoint local and remote replication to provide operational and disaster recovery for the Oracle Fusion Middleware and database environments. RecoverPoint provides application-consistent recovery points that can be utilized in response to a number of possible scenarios, enhancing the native availability of Oracle environments.


Audience The intended audience for this paper includes both storage administrators as well as database administrators seeking to understand best practices for failover and recovery of both Oracle Database Server and Oracle Fusion Middleware environments. The reader will establish an understanding of the importance of using an integrated solution to ensure effective recovery of all necessary data files to restart not only Oracle databases, but the applications they support. Detailed best practices are provided based upon EMC and Oracle joint testing, helping provide IT organizations with guidelines to develop a complete disaster recovery solution specific to their business.

RecoverPoint RecoverPoint is proven technology for high-availability Oracle environments with both local and remote protection across SAN storage with complete protection against many possible disaster scenarios. This type of environment provides resiliency against failures within the data center infrastructure. It can help improve recovery from a regional disaster, all with the added benefit of immediate and instantaneous application recovery.

Oracle products are inherently highly available and provide enterprise-class reliability without compromising security, performance, or scalability. To enhance the built-in availability features for Oracle, consider the following requirements for a data protection solution:

• Protection from infrastructure failure (such as a storage array or SAN switch) • Protection from local or regional disaster • Protection from data corruption Many companies are deploying continuous data protection (CDP) as a way to meet their recovery time objectives (RTO) and recovery point objectives (RPO). A true CDP implementation ensures that all changes to an application’s data are tracked and retained consistently. In effect, CDP creates an electronic journal of application data snapshots, one for every instant in time that a modification occurs.

Advantages of RecoverPoint RecoverPoint CDP preserves a record of the write transactions that take place within its environment, providing crash or application consistent recovery points. For local replication, RecoverPoint captures every write with I/O splitter technology (see the “RecoverPoint write splitters” section on page 33) and preserves them in a local journal; for remote replication, transactions are grouped based on user-specified policies, with significant write changes preserved in a journal at the DR site.

This preservation of writes ensures that if data is lost or corrupted, such as from a server failure, virus, Trojan horse, software errors, or end-user errors, it is always possible to recover a clean copy of the affected data. Another advantage for RecoverPoint is that this data recovery can be performed at either the local or remote locations. These recovery points can be immediately accessed and mounted back to production environments in seconds — much less time than is the case with disk-based snapshots, tape backups, or archives.

Local and remote recovery EMC RecoverPoint is a comprehensive data-protection solution providing concurrent local and remote (CLR) data protection. This integrates both CRR (remote) and CDP (local) replication, allowing users to recover applications to any point in time. The integration of local (CDP) and remote (CRR) replication protects data against catastrophic events that can bring entire data centers to a standstill. RecoverPoint delivers superior data protection by allowing both local and remote replication with no application performance degradation. As a result, organizations can deploy geographically dispersed data centers for maximum protection from local or regional failure or disaster.


RecoverPoint CDP, CRR and CLR use the same write-splitting methods, journaling technologies, and appliance platform. They both make use of consistency groups, which let the user define sets of volumes for which the inter-write order must be retained during replication and recovery. This ensures that data at any point in time will be fully self-consistent.

With RecoverPoint CDP, instead of compressing the data and sending over an IP network to a remote volume, it writes the data to a local journal and then to a local volume. As there is no IP network involved, and hence no latency concern, RecoverPoint CDP can synchronously track every write in the local journal and distribute the write to the target volume—all without impacting application-server performance.

RecoverPoint CRR transfers significant writes, based on an application’s RPO/RTO, to a remote site where they are saved in a history journal. Once the appliance receives the write, it will bundle this write up with others into a package. Redundant blocks are eliminated from the package, and the remaining writes are sequenced and stored with their corresponding timestamp and bookmark information. The package is then compressed, and an MD-5 checksum is generated for the package.

The package is then scheduled for delivery across the IP network to the remote appliance. Once received the remote appliance verifies the checksum to ensure the package was not corrupted in the transmission. The data is then uncompressed and written to the journal volume. Once the data has been written to the journal volume, it is distributed to the remote volumes, ensuring that write-order sequence is preserved.

RecoverPoint CRR enables the user to perform backups at the remote site, eliminating the need to take the production file systems offline.

With RecoverPoint data recovery can be performed locally and/or remotely by rewinding the target volumes back to a selected point in time by using earlier versions of data saved in the journal.

RecoverPoint CRR provided protection for Oracle Fusion Middleware data. It validated that an entire environment could be restored at a secondary or disaster recovery site, in the event of a major production site failure.

Replication modes When transferring data from the local site to the remote site, the RecoverPoint system automatically switches between replication modes on the fly to ensure that it is always using the method that best fits the current load conditions and the replication policy. The RecoverPoint system automatically switches between the following modes, according to load conditions: • Continuous synchronous mode • Continuous asynchronous mode • Snapshot mode RecoverPoint automatically uses the replication mode that is most effective for the current conditions, including the application load, throughput capacity, and replication policy. Regardless of the replication mode, RecoverPoint is unique in its ability to guarantee a consistent replica at the target side under all circumstances, and in its ability to retain write order fidelity in multi-host heterogeneous SAN environments. For more information regarding replication modes, please refer to the EMC RecoverPoint Release 3.1 Administrator’s Guide.

Oracle database protection with RecoverPoint EMC RecoverPoint supports both single instance and Oracle Real Application Clusters (RAC) for local and remote replication of the Oracle RAC SAN-attached volumes. The Oracle LUNs are grouped into a single consistency group, and replication sets are created to map the production LUNs to the remote and/or local copy LUNs. RecoverPoint then processes the consistency group based on the type of recovery required. The following are among the options Oracle provides for protecting its databases, which can be integrated with RecoverPoint replication.


• Application-consistent recovery from a shutdown (also known as “cold” backup)

The user creates a consistency group that represents the Oracle application. The consistency group contains all volumes for the application, including data files, online redo log files, configuration files, and optionally control files. This method produces a copy from which you can restore the database with 100 percent reliability. Because normal operations must be halted while the “cold” backup is being created, this method is not appropriate for systems that must operate on a 24 x 7 basis. In addition, any changes protected by RecoverPoint that are made before or after the “cold” backup will not be available as an application-consistent recovery point, but they can be recovered as a crash-consistent recovery point (see below). When the database is shut down, the user will create a RecoverPoint bookmark for the specific consistency group, identifying the image as a “cold” backup image. This bookmark can be used to identify a point-in-time recovery image that represents a fully restorable and restartable Oracle database image.

• Crash-consistent recovery during operations This process enables the creation of crash-consistent images without requiring system shutdown. This process is performed by default by RecoverPoint for all applications as part of the RecoverPoint write-splitting operations. As a write is issued to the production volumes, RecoverPoint splitters intercept it and send a copy of the write to the RecoverPoint appliance for further processing. These captured writes represent the on-disk consistent data, which is the same data that remains on external storage even when an application crashes. When Oracle is restarted from a server crash, an instance must first open the database and then execute recovery operations. The instance automatically uses the redo log to recover the committed data in the database buffers that was lost when the instance failed. Oracle also undoes any transactions that were in progress on the failed instance when it crashed, then clears any locks held by the crashed instance after recovery is complete. When Oracle is restarted from a RecoverPoint crash-consistent image, it will perform the same recovery procedures.

• Application-consistent recovery during operation (also known as “hot” or “fuzzy” backup) This process enables the creation of application-consistent images without requiring system shutdown. It is required that all data files belonging to the relevant tablespaces, archive log files, and control files are flushed from the server’s in-memory buffers to disk. To ensure that Oracle can recover from these images, Oracle must write additional information to the log file; information not required when crash-consistent images are sufficient. This feature in RecoverPoint requires that the user script several commands both to the Oracle Server and to the RecoverPoint appliance. The procedure entails placing the appropriate tablespace or database into backup mode (for example, ALTER TABLESPACE BEGIN BACKUP or ALTER DATABASE BEGIN BACKUP). When in “hot” backup mode, when a database block is modified, the entire block is written to the online redo logs. In normal operations, only the changed bytes are written. Also the data file headers are not updated with the SCN when a checkpoint is performed. Once Oracle backup mode is set, the script creates a RecoverPoint bookmark for the specific consistency group to identify the image as an application-consistent image. Archived redo logs can be used against this image.

Federated environments and consistency groups Federated environments are related applications that span multiple servers and storage arrays at the same site. Each application has its own RPO and RTO policies that govern the protection and recoverability of the application’s data. In RecoverPoint terminology, each application becomes a RecoverPoint consistency group with its own policy, journal, and replication set. For a successful recovery of the federated environment the customer must ensure that the individual consistency groups have a common recovery point across all of the applications.

To get a common recovery point, the user creates a “group set” that defines the RPO across all of the RecoverPoint consistency groups that make up the federated environment. This is done though the use of a common bookmark in each journal for the consistency groups that are part of the “group set.”


When the user recovers each of the consistency groups that make up the federated environment and selects the common bookmark, the user will ensure that all of the application’s data is recovered to the exact same point in time. This enables the data to be used for such advanced features as building a testing and development environment, creating a federated backup image, data mining, and so forth.

Use of federated consistency groups provides the following:

• Allows application recovery to be tiered by service level Multiple volumes per group Mixed recovery point objectives within the same infrastructure

• Provides independent replication controls Recover by group, either locally or remotely Start/stop by group

• Enables grouping of optimization Importance Resource usage Recovery point and recovery time objectives

• Each tier can have different service level agreements Consistency groups per tier Operational recovery of tier

• Enforces consistency across tiers Federated environments Recover to a known point for all applications Disaster recovery for tier or application Spans operating systems, applications, storage, and servers

• Enables advanced functions Full environment cloning Application upgrade testing Data mining Consistent production rebuild

The federated environment topology consists of three separate tiers: the web tier, the Oracle Fusion Middleware SOA Suite application tier, and the database tier. For successful recovery, consistency must be maintained between the middleware application tier and the database tier.


Use case objectives Objective Details

Validate the test system architecture deploying key Oracle and EMC technologies.

The archtecture consists of SAN-based storage with multiple Linux servers deployed in a production / DR site mode.

Utilize EMC RecoverPoint to protect all tiers, that is,web, Oracle Fusion Middleware, and database.

Describe how RecoverPoint is set up and administered to protect the Oracle WebLogic application server, configuration information, and the WebLogic persistent stores (transaction logs), the SOA Suite binaries and configuration, and the Oracle database.

Ensure a very small to zero RTO and a very small RPO. Detail how RecoverPoint CLR is used to create simultaneous local/remote replicas and bookmarks.

Provide no single point of failure. Accomplish with clustered WebLogic Application servers, EMC storage as well as highly available replication appliance (RPA) configurations.

Support the replication of WebLogic configuration and transaction logs (TLogs)

Achieve through EMC RecoverPoint CLR for local and remote replication.

Set up procedures to demonstrate planned switchover and switchback and unplanned failover and failback.

Show these procedures using EMC RecoverPoint.

Demonstrate ease of use in WebLogic and Oracle database environment failover and failback procedures.

Document these procedures with easy-to-follow failover and failback instructions.

Run the Oracle validation test suite, verifying failover data integrity.

Execute test program and procedures that validate RecoverPoint as a DR solution. This paper describes a test environment that is representative of DR protection of a production environment. The actual validation tests are not addressed here.

Oracle/EMC environment block diagram Figure 1 depicts the major components and relationships between the servers, and applications on the primary/production and disaster recovery sites. The configuration consists of a single web host and two application servers with Oracle Fusion Middleware. They are clustered using Oracle WebLogic Server clustering. EMC RecoverPoint protects all objects including the Oracle Home binaries, Oracle Fusion Middleware, and the single instance database.


Clients

App Host # 1 App Host # 2

Cluster

RecoverPoint Manager

RecoverPoint Appliance

App Host # 1

RecoverPoint (RP) CRR

RecoverPoint Appliance

BPEL+ ESB + OWSM

BPEL+ ESB + OWSM

Admin Server

RecoverPoint Manager

Admin Server

BPEL+ ESB + OWSM

BPEL+ ESB + OWSM

Cluster

App Host # 2

Web Host APACHE SERVER

Load Balancer

Standby Site



Load Balancer


Primary Site

Database Database Fusion Middleware & Logs

Data Guard Replication

Figure 1. Topology diagram of an SOA Suite Application and RecoverPoint use case environment


RecoverPoint validation architecture The RecoverPoint architecture for the middleware disaster recovery consists of four RecoverPoint appliances (RPAs) attached to both primary and secondary storage through Fibre Channel with two paths to either appliance. The connections are managed through a Brocade DCX and 7600 switch. They have an IP connection on a private subnet through a gigE dLink switch to simulate a WAN link; this is in addition to the management IP connection that runs through a Cisco Catalyst switch. Both storage arrays, the EMC

CLARiiON® CX4-480 and CX4-960, have the integrated RecoverPoint write-splitting technology. Each storage array has a storage group created specifically to mask the LUNs to the RecoverPoint appliances as well as the middleware hosts.

Figure 2. RecoverPoint use case architecture

All the hosts are zoned through the Brocade switches to the CX-480 and CX-960, using the switch explorer GUI. The RecoverPoint appliances were also zoned over to the storage in the same manner. For each LUN masked to the RecoverPoint appliances, a replication set is created linking the source LUN to a target LUN. These replication sets are all placed within the middleware consistency group.

The CX splitter is integrated with each storage processor (SP) of the CLARiiON array and this will send one copy of the write to the storage array and the other to the RecoverPoint appliance.

The following types of storage volumes are required for RecoverPoint configuration:

• Repository volume: This volume holds the configuration and marking information during replication. At least one repository volume is required per site and is accessible from all the RPAs at the site.

• Journal volume: This volume is used to store all the modifications. The application-specific bookmarks and timestamp details are written to the journal volume. The size of the journal depends on the write activity to the protected LUN(s) and the RPO required for the data. Best practices information on the sizing and configuration of journal volumes is available in the EMC RecoverPoint Installation Guide.


• Replication set: The association created between the source volume and the local and/or remote target

volumes is called the replication set. A consistency group contains one or more replication sets. • Consistency group: The logical grouping of replication sets identified for replication is called a

consistency group. Consistency groups ensure that the updates to the associated volumes are always consistent with write-order preserved and that they can be used to restore the database at any point of time.

The following shows the Middleware consistency group and its assigned replication sets from the RecoverPoint Management Application GUI. The names of the volumes in the Middleware and DR columns are derived from the “friendly” names of the LUNs returned by the CLARiiON array during the SCSI discovery operations. The number in parenthesis is the CX LUN ID number.

Figure 3. Consistency group and replication set definitions


The following RecoverPoint Management Application GUI screenshot shows the Middleware consistency group storage volumes replicated by RecoverPoint (defined by replication sets).

Figure 4. Storage volumes in replication sets

CLARiiON array-side preparations The integrated RecoverPoint write-splitting technology is enabled on each storage processor (SP) of the CLARiiON array. The integrated write-splitter will intercept every write to a protected LUN and will send a copy of the write to the RecoverPoint appliance and send the original to the protected LUN. This setup has a CX4-480 as primary storage and a CX4–960 at the DR site. RecoverPoint 3.1 is installed and configured to utilize the integrated RecoverPoint write-splitting technology in the CX4 arrays.


System configuration and storage

Hardware configuration Table 1. Server types used

Dell PowerEdge 2950 Dell PowerEdge 900

2 Quad-Core CPUs,

2.50 GHz, 16 GB RAM,

Brocade 8 GB HBAs

4 Quad-Core CPUs,

2.93 GHz, 32 GB RAM,

Brocade 8 GB HBAs

Table 2. Fibre Channel switch types

Primary DR

Brocade DCX Brocade 7600

Table 3. Storage types used

Primary DR

CLARiiON CX4-480 FLARE 28

CLARiiON CX4-960 FLARE 28

Table 4. Server functions and types

Server function Server type

Production Site – Apache Web Server Dell 2950

Production Site – Oracle WebLogic Server Cluster 1

Dell 2950

Production Site – Oracle WebLogic Server Cluster 2

Dell 2950

Production Site – Oracle Database Server Dell 900

DR Site – Apache Web Server Dell 2950

DR Site – Oracle WebLogic Server Cluster 1 Dell 2950

DR Site – Oracle WebLogic Server Cluster 2 Dell 2950

DR Site – Oracle Database Server Dell 900


Software configuration Table 5. Software versions and configuration

Software Version Configuration

OEL Enterprise Linux Server (Carthage)

Release 5.2 x86_64

EMC PowerPath® 5.1 SP 2

EMC RecoverPoint CLR 3.1

Oracle Database 10g Release 2 10.2.0.1 Linux X86_64

Oracle Database Server Patch Set 10.2.0.4 Patch Set 6810189

Linux X86_64

Oracle Enterprise Manager 10g Grid Control

10.2.0.4

Oracle Enterprise Manager 10g Grid Control Agent

10.2.0.4 Linux x86_64

Oracle WebLogic Server 9.2 MP3 Linux x86

Oracle Application Server SOA Suite 10g Release 3

10.1.3.1 Linux x86

Oracle Application Server Patch Set

10.1.3.4 Patch Set 7272722

Oracle SOA Suite 10.1.3.4 for WebLogic Server 9.2 MP3 Patch

Patch 7490612

Oracle Patch – Resolve Prereq Installation Issue

10.1.3.1 Patch 633905

Linux x86_64

Apache HTTP Server 2.2.3

Oracle JDeveloper 10g Release 3 Studio Version 10.1.3 Windows XP – Service Pack 2

Apache ANT 1.7.1


Oracle Fusion Middleware, web tier, and Database volume layout Table 6. Volume layout

Volume name

Tier Size Mounted on nodes

Mount point Notes/Comments

VolWeb Web Tier

20G webhost /u01/app/oracle Volume for Apache Install

VolAdmin App Tier

20G apphost1 /u01/app/oracle/wls/soaDomain/admin

Volume for Admin server Instances

VolWLS1 App Tier

20G apphost1 /u01/app/oracle/wls/soaDomain/mng1

Volume for Managed Server Instance

VolWLS2 App Tier

20G apphost2 /u01/app/oracle/wls/soaDomain/mng2

Volume for Managed Server Instance

VolData App Tier

20G apphost1, apphost2

/u01/app/oracle/data Volume for TLogs and JMS Data

VolOrcl1 App Tier

20G apphost1 /u01/app/oracle/product Volume for binaries, both Oracle and WebLogic

VolOrcl1 App Tier

20G apphost2 /u01/app/oracle/product Volume for binaries, both Oracle and WebLogic

Oradata_1 DB Tier

100G dbhost /u01/app/oracle Volume for Oracle binaries and Flash Recovery Area

Oradata_2 DB Tier

100G dbhost /u01/oradata/orclsoa/cg1_dbf_undo Database Files and Undo

Oradata_3 DB Tier

50G dbhost /u01/oradata/orclsoa/cg2_redo Database Online Redo Logs

Oradata_4 DB Tier

100G dbhost /u01/oradata/orclsoa/cg3_arch_ctl Database Archive Logs and Controlfiles

Journal RP 50G RP appliance

Raw Volume - Fusion Middleware Consistency Group Production

RecoverPoint metadata


Raw Volume - Fusion Middleware Consistency Group CRR Replica



Raw Volume - Database Consistency Group, cg1_dbf_undo Production



Raw Volume - Database Consistency Group, cg1_dbf_undo CRR Replica


Journal RP 50G RP Raw Volume - Database Consistency RecoverPoint


appliance Group, cg2_redo Production metadata


Raw Volume - Database Consistency Group, cg2_redo CRR Replica



Raw Volume - Database Consistency Group, cg3_arch_ctl Production



Raw Volume - Database Consistency Group, cg3_arch_ctl CRR Replica


Please note the VolData volume is mounted simultaneously on both application hosts. This volume contains the WebLogic persistent file-based store. File-based stores must be configured on shared storage. The OCFS2 clustered file system is configured for the shared storage.

All server root volumes are installed on local disk. Although not used in this validation, RecoverPoint does support the replication of root volumes installed in the SAN (boot from SAN).

RecoverPoint CRR consistency groups For this use case three additional consistency groups for the database tier were created. The Oracle Fusion Middleware tier consists of one consistency group as previously configured. The white paper Disaster Recovery of Oracle Fusion Middleware with EMC RecoverPoint has further detail.

The Oracle Fusion Middleware has two types of data that are being replicated. The first is the SOA Suite, WebLogic binaries, and configuration files. The second is the persistent file-based store used for the Java Message Services (JMS) and Transaction Logs (TLogs). The persistent file-based store is used to recover transactions.

The RecoverPoint consistency groups used for the database tier will be provisioned for 1) data files and undo logs, 2) online redo logs, and 3) archived redo plus control files. This will provide for the various strategies of disaster recovery according to the business needs of the organization, that is, application-consistent recovery, crash-consistent recovery, and crash-and-application consistent recovery.

The online redo logs change frequently. In a RecoverPoint configuration, placing the online redo logs in a separate, lower-priority consistency group and giving priority to the data and archive redo log replication sets (crash-consistent strategy) may improve replication performance and slightly decrease time between snapshots (point-in-time recovery objective). The actual impact for all disaster recovery strategies depends on the bandwidth and the percentage of writes being sent over the link to the replica site. Please refer to the Replicating Oracle with EMC RecoverPoint Technical Notes.

Each consistency group has settings and policies to include configuration group name, preferred RPA, and reservation support; and policies, such as compression, bandwidth limits, and maximum lag, which govern the replication process. In the configuration described above, the Lag time will need to be defined. This is the maximum offset between writing data to production storage and writing it to the RPA or journal at the replication site. The EMC RecoverPoint Release 3.1 Administrator’s Guide has further detail on the settings and policies.

RecoverPoint group sets and parallel bookmarks RecoverPoint group sets are used throughout a federated environment. Federated environments are related applications that may span multiple servers and storage arrays. Each application has its own RPO and RTO policies that govern the protection and recoverability of the application’s data. After creating each consistency group with its appropriate settings and policies, to ensure that the individual consistency groups have a common consistency point across all of the applications, create a group set.

The group set allows you to automatically bookmark a set of consistency groups so that the bookmark represents the same recovery point of each consistency group in the group set. This allows you to define


consistent recovery points for consistency groups that are distributed across different RPAs. The automatic periodic bookmark consists of the name you specified for the group set and an automatically incremented number. Numbers start at zero, and are incremented up to 65535, then begin again at 0.

The same bookmark name is used across all consistency groups. To apply automatic bookmarks, the sources must be at the same site (replicating in the same direction) and transfer must be enabled for each consistency group included in the group set. The Group Set Details dialog box in the RecoverPoint Manager Application allows you to create, edit, or remove group sets.

When creating the group set, enter a name for the automatic bookmarks. Select the consistency groups to be in the group set, and specify the bookmarking frequency. This will enable the parallel bookmarks. It is recommended that the interval between automatic bookmarks not be less than 30 seconds.

If you prefer to take a bookmark at a specific time other than the automated times, or choose to manually take the parallel bookmarks, this can also be done through the RecoverPoint Manager Application.

In the Navigation pane, select consistency groups. In the component pane, select all the consistency groups to bookmark simultaneously. All selected consistency groups must be enabled and transfer must be active. Click the parallel bookmarks icon in the upper right corner of the component pane. When prompted, enter the name of the bookmark. Please do not use the name “latest” as it is a reserved word in RecoverPoint.

The group set can be perceived as a single entity, but each consistency group functions as a separate unit within the group set. The user recovers each of the consistency groups that make up the federated environment and selects the common bookmark. This will ensure all of the application’s data is recovered to the exact same point in time. This can be executed through the RecoverPoint Manager Application or using the command line interface (CLI).

Maintaining the autonomy of the consistency group in a group set provides the flexibility for each application to maintain its own policies and settings to govern the protection and recoverability of the application’s data. It enables the data to be used for such advanced features as building a testing and development environment, creating a federated backup image, and data mining.

OCFS2 configuration

High availability for persistent stores The WebLogic application servers are usually clustered for high availability. For the local site high availability of the SOA Suite Topology, a persistent file-based store is used for the Java Message Services (JMS) and Transaction Logs (TLogs). This file store needs to reside on shared disk that is accessible by all members of the cluster, that is, by apphost1 and apphost2. The persistent file-based store can be migrated along with its parent server as part of the whole server migration feature that provides both automatic and manual migration. There are two methods for configuring a synchronous write policy: the Cache-Flush and the Direct-Write policies. The Cache-Flush policy improves performance, but the downside is possibly losing sent messages or generating duplicate messages in the event of an operating system crash or hardware failure. This is due to the fact that transactions are complete as soon as the writes are cached in memory, instead of waiting for acknowledgement that the writes are written to disk.

Use high-availability storage for state data The server migration process moves or “migrates” services. Some state information associated with the work in process at the time of failure is persisted to storage. To ensure high availability, it is critical that such state information remains available to the server instance and the services it hosts after migration. It should be stored in a shared storage system that is accessible to any potential machine to which a failed migratable server might be migrated. For highest reliability, use a shared storage array solution (like EMC

CLARiiON) that is itself highly available and a SAN designed for high availability.


If independent local file systems resided on a shared LUN, there would be no means of cache synchronization, and the file systems would eventually corrupt each other. Therefore, the shared storage solution in a SAN as deployed in this white paper uses the Oracle Cluster File System (OCFS2) as the host-based clustered or shared file system technology. OCFS2 is a symmetric shared disk cluster file system that allows each node to read and write both metadata and data directly to the SAN. A shared storage solution in a SAN as described in this paper uses a host-based clustered or shared filesystem technology. Oracle Cluster File System (OCFS2) is used here, but any shared filesystem technology can be used such as Red Hat's GFS, Veritas VxCFS, or IBM GCFS.

Network configuration The EMC RecoverPoint system has been designed as a secure platform for CDP (local), CRR (remote), and CLR (local and remote replication). EMC has invested in ensuring security for all aspects of its RecoverPoint system, including the operating system, networking, and RecoverPoint software. Security settings are divided into the following categories: • RecoverPoint appliance (RPA) operating system and networking • Access control settings to limit access by end users or by external product components • Log settings related to the logging of events • Communication security settings related to security for the product network communications • Data security settings available to ensure protection of the data handled by the product

Secure serviceability is available to ensure control of service operations performed on the products by EMC

or its service partners. Other security considerations list known issues, including false positive findings that appear when scanning the product for vulnerabilities. For further discussion and implementation detail, please refer to EMC RecoverPoint Release 3.1 and Service Pack Releases Security Configuration Guide.

Planning for disasters and planned downtime

Initiate the replication process RecoverPoint is an intelligent data protection and recovery solution that runs on out-of-band appliances attached to the SAN and the IP network. RecoverPoint uses your enterprise’s existing network and storage systems. To prepare for installation, you should be familiar with how the RecoverPoint system integrates with your existing systems. In preparation for RecoverPoint installation:

1. The RPAs connect to the hosts and storage subsystems using a Fibre Channel SAN. Before installing RecoverPoint, these subsystems should be in place.

2. RPAs are linked to the WAN interface using an Ethernet/IP connection (eth0).

3. Before you install RPAs and define consistency groups, ensure that sufficient volumes are

available on the SAN-attached storage at each site for use by RecoverPoint. As part of the


installation process, you must create a repository volume and journal volumes at both the primary and secondary sites. The journal is maintained on one or more SAN-attached storage volumes for each copy of a consistency group; that is, the production copy, together with a local copy or a remote copy, or both. To configure a consistency group, you must designate one or more volumes on the production storage to be replicated, together with corresponding volumes on the copy (or copies). RecoverPoint operation requires proper storage mapping (LUN masking) configuration.

4. The RecoverPoint system uses the Network Time Protocol (NTP) to synchronize the clocks across

all of the machines that are attached to a given installation. It is highly recommended that you configure an external NTP server, which runs on Linux machines, for use by the RecoverPoint system prior to RecoverPoint installation.

5. Zoning on the Fibre Channel switch must be determined. It allows for communication between

the host, storage, and appliances. Zoning instructions vary according to the type of HBA ports installed in your RPAs. Without the proper zoning, replication cannot be performed properly.

6. The RPAs should be installed independently at the primary and secondary sites.

RecoverPoint software installation and configuration RecoverPoint is installed with a GUI wizard that helps you install new RecoverPoint clusters from a single management point. At the end of the wizard, you will be able to start building a replication configuration. Before you begin using the RecoverPoint Installer, ensure you have completed the following operations:

1. Identified the type of installation (one site or two sites).

2. Confirmed that all RPAs are available at the site(s).

3. Confirmed that at least two ports are available on each of two fabrics for each RPA.

4. Completed the integration with installed networks and storage.

5. Completed the Customer Site Planning Sheet.

6. Unpacked and physically installed the RPAs.

7. Assigned Management IP addresses for the RPAs (done when logging in as the user “boxmgmt” with the password “boxmgmt”). The IP addresses are used by the RecoverPoint Installer to complete the installation.

For detailed information, please refer to the EMC RecoverPoint Release 3.1 Installation Guide. After completing system installation for the first time and verifying that all clocks are synchronized, verify that you can access the management interface using both the RecoverPoint Management Application GUI and the CLI.

Starting replication It is assumed that host-based, fabric-based, and array-based splitters have been installed as needed. If not, please refer to the EMC RecoverPoint Release 3.1 Installation Guide. Before you begin, add splitters to the RecoverPoint system.

Create the consistency groups using the Add New Group Wizard. Define the settings and policies for the consistency group.


Configure the copies. Enter the values for the General Settings, Protection Settings and Advanced Settings for the production, local, and remote replicas. If you are using the Create New Consistency Group Wizard, the Production Copy Settings dialog box appears. In the Replication Set and Journal Volumes Configuration dialog box, click the Add New Replication Set button to define a new replication set. A replication set consists of a source (normally production) volume and a corresponding volume for each replica. Each consistency group contains as many replication sets as there are source volumes in the configuration group. Add the volumes to the splitters that have been installed. When the volumes are added, they will be automatically attached to all of the splitters that have access to that volume. Now that the configuration groups and their journals are defined and configured, splitters have been added, and volumes attached to splitters, enable the groups. Enabling the groups will initiate the transfer/replication/sync of the production volumes with the replica/disaster recovery site volumes. When a consistency group is initialized for the first time, the RecoverPoint system must complete full synchronization of all designated volumes. The volumes at the local and remote site can be initialized while host applications are running.

Alternatively, the current production middleware installation can be backed up, and manually transferred to the remote site. This is possible, because the RecoverPoint system can efficiently determine which blocks are different between the production and replica copies. It sends only the data for those blocks to the replica storage, as the initialization snapshot. For detailed procedures and further explanation, please refer to the EMC RecoverPoint Release 3.1 Administrator’s Guide.

Switchover procedures Switchovers are planned operations for the purpose of testing and validation that result in smooth transition of services and applications from one site to another. A trigger may be a maintenance window or to comply with regulatory requirements to validate disaster recovery functionality. Switchovers can provide a mechanism to establish, test, and prove SLA, RPO, and RTO requirements.

During a switchover, the current production site becomes the disaster recovery site, and the disaster recovery site becomes the current production site.

The initial state of the production site is presumed to be up and functioning. The procedures begin with the shutdown of the production site.

The procedures to execute the switchover are as follows.

1. The decision is made to initiate switchover to the disaster recovery site, and all participants in this process are advised.

2. Shut down the Oracle Fusion Middleware, that is, SOA Suite and WebLogic.

a) Shut down the WebLogic Managed Servers.

Log in to the WebLogic Console. Choose SOADomain > Control. Select the managed servers, and click the Shutdown tab.

Choose the option Force Shutdown Now.

b) Shut down the WebLogic processes on the application servers.

Log in to the WebLogic Administration Server application host. Using the command line, manually shut down the Node Manager process first and then shut down the Administration Server process.


Log in to the other nodes in the cluster and manually shut down the Node Manager process. In this configuration, the WebLogic Administration Server is only configured on one server.

3. Log in to the webhost server(s). Shut down the Apache httpd processes.

4. Unmount the file systems on all of the application and webhost servers.

5. Log in to the database server.

a) Cleanly shut down the database and Listener. If the database is managed by Oracle Enterprise Manager 10g Grid Control, then log in to Oracle Grid Control and shut down the database and Listener. If not, use any variation of the standard Oracle commands, that is, “shutdown immediate” and “lsnrctl stop”.

b) Unmount the database file systems.

6. Perform any network, DNS updates, or modifications to /etc/hosts or DNS if necessary at the disaster recovery site.

7. Log in to the RecoverPoint Management Application.

a) Confirm current image selection on the production site:

All applications are now shut down. Take a parallel bookmark of the production site. The parallel bookmark is needed to identify the point across all of the consistency groups that represents the point in time the shutdown has completed. It causes any data held at the production site to be flushed to the remote site. Therefore if a recovery is needed of the production site prior to switchover, the parallel bookmark is guaranteed to have the latest data prior to the switchover. For detailed procedures to create the parallel bookmark, please refer to the EMC RecoverPoint 3.1 Administrator’s Guide, in the section, “Applying bookmarks to multiple groups simultaneously.”

A journal entry reflecting the parallel bookmark will be entered into each consistency group’s journal. In this configuration, there are five consistency groups, and therefore respectively, five journal entries.

To view the bookmark for each consistency group, in the Navigation pane, click on the consistency group name. Then choose the production site and click the Journal tab. When you see the journal entry for the bookmark, you can be assured that all the data has flowed through the pipe and has been replicated to the DR site.

b) On the DR site for each consistency group in the group set:

Enable access to the replicated images and applications at the DR site.

Click on the first icon under the name of the DR site. Choose Enable Image Access.

Choose Select an image from the list. From the list that appears, select the named parallel bookmark.

Choose the type of access mode. The production site will be recovered from the DR site. Therefore, select Virtual access with instantaneous access to the image, and Roll image in the background to enable the recovery process.

Confirm the current image selection


a) Mount the database file systems.


b) Start the database and Listener. If the database is managed by Oracle Enterprise Manager 10g Grid Control, then log in to Oracle Grid Control and start the database and Listener. If not, use any variation of the standard Oracle commands, that is, “startup” and “lsnrctl start”.

9. Mount the file systems on all the application and webhost servers.

10. Log in to the webhost server(s). Start the Apache httpd processes.

11. Start the Oracle Fusion Middleware, that is, SOA Suite and WebLogic.

a) Start the WebLogic processes on application servers.

Log in to the WebLogic Administration Server application host. Using the command line, manually start the Administration Server process first and then start the Node Manager process.

Log in to the other nodes in the cluster and manually start the Node Manager process. In this configuration, the WebLogic Administration Server is only configured on one server.

b) Start the WebLogic Managed Servers.

Log in to the WebLogic Console. Choose SOADomain > Control. Select the managed servers, and click the Start tab.

12. The site is ready to perform work. Applications are started, and database has been switched over.

Switchback procedures Testing and validation of the disaster recovery plan have been completed. SLA, RPO, and RTO requirements have been verified, and compliance with regulatory requirements has been met and proven. Planned maintenance is done. The next tasks are to prove the procedures to switch back from the disaster recovery site to the primary site and in this test scenario, to reserve all changes and propagate to the primary site.

During a switchback, the disaster recovery site, now acting as the primary site, returns to its initial function as the DR site.

The initial state of the DR site is up and functioning. The production site is down. We begin by shutting down all processes on the DR site in preparation to recover the production site from the DR site images with RecoverPoint.

The procedures to execute the switchback are as follows.

1. The decision is made to initiate switchback to the production site, and all participants in this process are advised.

2. Shut down the Oracle Fusion Middleware, that is, SOA Suite and WebLogic.

a) Shut down the WebLogic Managed Servers.

Log in to the WebLogic Console. Choose SOA Domain > Control. Select the managed servers, and click the Shutdown tab.

Choose the option to Force Shutdown Now.

b) Shut down the WebLogic processes on the application servers.

Log in to the WebLogic Administration Server application host. Using the command line, manually shut down the Node Manager process first and then shut down the Administration Server process.


Log in to the other nodes in the cluster and manually shut down the Node Manager process. In this configuration, the WebLogic Administration Server is only configured on one server.

3. Log in to the webhost server(s). Shut down the Apache httpd processes.

4. Unmount the file systems on all of the application and webhost servers.


a) Cleanly shut down the database and Listener. If the database is managed by Oracle Enterprise Manager 10g Grid Control, then log in to Oracle Grid Control and shut down the database and Listener. If not, use any variation of the standard Oracle commands, that is, “shutdown immediate” and “lsnrctl stop”.

b) Unmount the database file systems.


7. Log in to the RecoverPoint Management Application.

a) On the DR site:

For each consistency group in the group set, to switch back to the production site, choose the option Recover production from the DR site. RecoverPoint will automatically take a snapshot of the latest image. The journal entry for the bookmark will be referred to as the “Pre-replication Image”. This is the image restored on the production site.

When the Recover production option is chosen, there will be several questions asking whether you are sure you want to perform the production restore, and do you want to continue. Respond Yes to these questions.

During the initial phases of transfer, the connection will be paused for reconfiguration. This can be seen in the Component pane of the RecoverPoint Management Application. When the value Transfer changes from Paused to Active, you can now move to the production site to complete the switchback recovery.

b) On the production site:

Replication will continue from the DR site to the production site, until all data is transferred. The amount of time to complete transfer is relative to the amount of changes and bandwidth. In the Component pane for the selected consistency group the image status of the production site will be “Distributing Pre-replication image” and the role will be “Production (being restored)”.

When the Pre-Replication image is recorded on the production site, all data has been replicated, and syncing completed. To verify the status of the transfer, click on the entry for the production site under each consistency group. Choose the Journal tab. The bookmark entry will indicate the replication status. When it states “Synchronization completed (Primary)”, replication is complete and you can continue with the next step.

For each consistency group, enable image access. Choose logged access (physical) when recovering the site on a permanent basis. Virtual images are temporary, as their name indicates.

After confirming the image, choose Resume production for each consistency group on the production site. You will be asked if you choose to continue with this action. It will be noted that there will be a pause while reconfiguration occurs. The response should be Yes.


When “Resume production” is completed at the production site, note the direction of the replication flow will change and revert back, originating from the production site to the DR site for each consistency group.

The volumes are now visible on the production site and are mountable as read/write. The RecoverPoint splitter is enabled, and all new writes will be sent to the appliance.

Although not required, best practice would recommend a parallel bookmark be created for the group set. This should be implemented before restarting the applications on the production site. This will provide an audit trail and a bookmark of all applications at a consistent point in time, in case there is an unexpected situation and you must again switch over immediately to the DR site.


a) Mount the database file systems.

b) Start the database and Listener. If the database is managed by Oracle Enterprise Manager 10g Grid Control, then log in to Oracle Grid Control and start the database and Listener. If not, use any variation of the standard Oracle commands, that is, “startup” and “lsnrctl start”.

9. Mount the file systems on all the application and webhost servers.

10. Log in to the webhost server(s). Start the Apache httpd processes.

11. Start the Oracle Fusion Middleware, that is, SOA Suite and WebLogic.

a) Start the WebLogic processes on the application servers.

Log in to the WebLogic Administration Server application host. Using the command line, manually start the Administration Server process first and then start the Node Manager process.

Log in to the other nodes in the cluster and manually start the Node Manager process. In this configuration, the WebLogic Administration Server is only configured on one server.

b) Start the WebLogic Managed Servers.

Log in to the WebLogic Console. Choose SOA Domain > Control. Select the managed servers, and click the Start tab.

12. The production site is recovered. Data is being replicated to the DR site.

Failover procedures The extent of the disaster and anticipated length of time at the disaster recovery site will directly affect which types of failover procedures are required. If the failover was due to a catastrophe, such as fire, flooding, or earthquake, then migration to the disaster recovery site is the likely scenario. In this case, the personality of the sites changes. The disaster recovery site becomes the permanent production site until the previous production site is rebuilt or repaired.

With this type of failover, when the production site becomes available, a resynchronization of all data and applications will be executed. For the RecoverPoint procedures to execute a migration please refer to the EMC RecoverPoint 3.1 Administrator’s Guide.

In the following scenario, the process addresses failover due to temporary loss of site, or unplanned downtime, and the system crashes. The procedures are very similar to those detailed in the previous “Switchover procedures” and “Switchback procedures” sections. The major difference is we are beginning with no access to the production site, and no ability to gracefully shut down all applications and cleanly switch over the database.


The procedures to execute the failover are as follows:

1. Log in to RecoverPoint Management Application. Enable access to the replicated images, as per steps 7b in the “Switchover procedures” section that starts on page 22. The wizard will ask which image to choose. For this exercise, we choose the named parallel bookmark image taken prior to switchover from the list of bookmarks.

2. Start the database and Listener. If the database is managed by Oracle Enterprise Manager 10g Grid Control, then log in to Oracle Grid Control and start the database and Listener. If not, use any variation of the standard Oracle commands, that is,“startup” and “lsnrctl start”.

3. Continue with steps 9 through 11 of the “Switchover procedures” section, that is, mount and start the Apache HTTP application.


5. The site is ready for work.

Failback procedures The systems on the production site are now available and a decision is made to return to the production site. The applications and database at the disaster recovery site must be shut down, failed back, or re-instantiated and restarted.

The procedures to execute the failback are as follows.

1. Shut down the WebLogic Managed Servers, WebLogic processes, and Apache httpd services, and unmount the file systems, as per steps 2 through 4 of the “Switchback procedures” section of this paper starting on page 24.

2. Cleanly shut down the database and Listener. If the database is managed by Oracle Enterprise Manager 10g Grid Control, then log in to Oracle Grid Control and shut down the database and Listener. If not, use any variation of the standard Oracle commands, that is, “shutdown immediate” and “lsnrctl stop”. Unmount the database file systems.

3. In this implementation, the failover was temporary. Therefore, to fail back, log in to the RecoverPoint Management Application and repeat steps 7a and 7b of the “Switchback procedures” section.

4. There is no direct relationship between having to re-instantiate or recover the database and RecoverPoint. It could be possible due to the bandwidth and the size of the database that it would be better to recover from backup, and then sync only the changes with RecoverPoint. To implement this procedure, please refer to the EMC RecoverPoint Release 3.1 Administrator’s Guide. If re-instantiation is not required, then mount the database file systems. Start the database and Listener. If the database is managed by Oracle Enterprise Manager 10g Grid Control, then log in to Oracle Grid Control and start the database and Listener. If not, use any variation of the standard Oracle commands, that is, “startup” and “lsnrctl start”

5. After completion of these steps, the original production site will be available to resume work. To start the applications repeat steps 9 through 11 of the “Switchover procedures”.

6. Perform any network, DNS updates, or modifications to /etc/hosts or DNS if necessary at the production site. The production site is recovered. Data is being replicated to the DR site.


General recommendations

Setting snapshots or manual bookmarks based on requirements The RPO policy for the Oracle Fusion Middleware binaries has been set based on size, number of writes, and time, with a configured lag time of 12 hours. The lag time of 12 hours was decided upon based on the knowledge that binaries have few expected modifications. The way to think about your policy is that you are guaranteed a recovery point within 12 hours.

The binaries are scheduled to be patched. We would like to ensure the patched binaries are replicated prior to the configured lag time after the application of the patch. The concern is if an event occurs that causes failover to the DR site, then the correct version of the code will be available.

To ensure the configuration changes are replicated, a bookmark is created. A bookmark is a named snapshot (image) that uniquely identifies the image at a point in time. The bookmark creates a transactional consistent snapshot that is transferred to the replica. When the bookmark is created it forces all data held at the production side in the RPA’s buffers to be flushed to the remote site. At the remote site a corresponding bookmark is recorded when replication is completed.

In this situation, it is best practice to create a bookmark prior to patching the binaries. If a recovery of the binaries prior to patching is needed, the “pre-patch” bookmark will guarantee the latest data prior to the patch.

After patching the binaries, create a “post-patch” bookmark, so a recovery using this bookmark is guaranteed to have the latest data image after the patch.

By creating both a “pre-patch” and “post-patch” bookmark, it becomes simple to choose either bookmark, depending on the point-in-time recovery required.

To create the bookmark, in the Navigation pane of the RecoverPoint Management Application, select the consistency group. Verify on the Status tab that it is active. Click the Bookmark button and enter a descriptive name.

Images’ snapshots will continue independent of the manually initiated bookmark per the RPO set policy.

Periodic DR testing To ensure governance, regulatory requirements, and failover to the disaster recovery site will be successful in times of catastrophe, it is a good practice to verify the replicas can be used to restore data, recover from a disaster, or seamlessly take over. In most cases, while testing a replica, applications can continue to run on the production servers, and replication can continue. The writes will be stored in the replica journal until testing is completed.

Upon completion of testing, write access to the replica is disabled, which results in any writes made during testing rolled back by RecoverPoint. However, any concurrent writes from the production applications will be automatically distributed from the journal to the replica. This entire process can be completed without application downtime and without loss of data at the replica.

In the RecoverPoint Management Application at the DR site:

1. From the Image Access menu, select Enable Image Access. If you are only testing the image, and do not expect a high rate of modification/change to the data, selecting Virtual Image Access without Roll Image in Background is appropriate. If you expect to do more testing/forensics, over an extended period of time or need maximum performance while testing, select Logged Image Access (physical). Virtual access is instantaneous while physical access will require more time before the image is available for use. For further discussion of the differences in the image access modes, please refer to the “Appendix” section of this paper.


2. At the host, mount the replica volume you wish to access. If the volume is in a volume group managed

by a logical volume manager, import the volume group.

3. If desired, run “fsck” (“chkdsk” on Windows) on the replica volumes. This is optional.

4. Access the volumes and test as desired.

5. When testing is completed, unmount the replica volumes from the host. If using logical disk management, deport the volume groups. Then select Disable Image Access at the replica. The writes to the replica will automatically be undone.

Event notification Complete event logging is provided for all RecoverPoint management operations and status changes.

Events are stored within the RecoverPoint System Log, which is accessible from the management application. Event logging includes auditing information such as commands, command errors, events, and the System Log. Additionally, RecoverPoint also supports the following types of event notification: e-mail, SNMP, Syslogs, System Reports, and System Alerts.

General I/O and sizing The transfer rate of a single RecoverPoint appliance (RPA) is approximately 60 MB per second per RPA.

Iostat can be used to measure the data change rate. This data is relevant in determining the number of RPAs needed to meet RTO and Service Level Agreements. It is useful in determining the size of the journals and retention time, bandwidth, and degree of compression.

Group sets should be configured for consistency groups that are dependent on one another or that must work together as a single unit, that is, in a federated environment. A group set provides the capability to apply parallel bookmarks at a user-defined frequency. In this implementation, the data is being replicated from both the web server and application hosts. On the application hosts, the rate of change varies for different volumes. The WebLogic persistent store must be replicated at a higher rate than the SOA and WebLogic binaries and configuration data need to be transferred. To implement group sets, please refer to the EMC RecoverPoint 3.1 Administrator’s Guide.

Conclusion Enterprise Oracle deployments need protection from unforeseen disasters and natural calamities. Oracle provides Data Guard as a technology to remotely replicate Oracle database files synchronously or asynchronously to allow for recovery of the Oracle database. However, protecting the database alone is not enough to protect the business itself. Full application level recovery is required to bring the business back to a production-ready state. To that end, Oracle works with partners to validate enterprise replication technologies to protect Fusion Middleware environments on which Oracle enterprise business applications run. This white paper summarizes the results of a joint EMC-Oracle engineering effort to utilize EMC RecoverPoint for local and remote replication of Oracle Fusion Middleware and Oracle Database Server information. It highlights key concepts and setup and administration examples of how to deliver application-aware recovery to specific points in time and provide continuous data protection , while also incorporating additional features such as bandwidth compression to reduce overall TCO.


References and resources

Oracle OCFS2 A Cluster File System for Linux, User’s Guide for Release 1.4 http://oss.oracle.com/projects/ocfs2/dist/documentation/v1.4/ocfs2-1_4-usersguide.pdf Oracle Cluster File System 2 (OCFS2) User’s Guide OCFS2 – Frequently Asked Questions http://oss.oracle.com/projects/ocfs2/documentation/v1.2/ Oracle WebLogic Server 9.2 documentation on the Oracle website http://e-generation.beasys.com/wls/docs92/admin.html SOA Architect Center page on the Oracle website http://www.oracle.com/technology/tech/soa/index.html Oracle Data Guard Concepts and Administration 10g Release 2 (10.2) http://download.oracle.com/docs/cd/B19306_01/server.102/b14239/toc.htm

EMC The following white papers can be found EMC.com. For a selection of other EMC RecoverPoint white papers, go to our resource library. • Introduction to EMC RecoverPoint 3.1: New Features and Functions http://www.emc.com/collateral/software/white-papers/h2781-emc-recoverpoint-3-new-features.pdf • Using EMC RecoverPoint Concurrent Local and Remote for Operational Disaster Recovery http://www.emc.com/collateral/software/white-papers/h4175-recoverpoint-concurrent-local-remote-oper-disaster-recovery-wp.pdf • EMC RecoverPoint Family Overview http://www.emc.com/collateral/software/white-papers/h2346-recoverpoint-ov.pdf The following are available on Powerlink, EMC’s password-protected website for customers and partners: • Enhancing Oracle Database Recovery with EMC RecoverPoint—Applied Technology\ • Disaster Recovery of Oracle Fusion Middleware with EMC RecoverPoint • Replicating Oracle with EMC RecoverPoint Technical Notes • EMC RecoverPoint Release 3.1 Administrator’s Guide • EMC RecoverPoint Release 3.1 and Service Pack Releases Security Configuration Guide • EMC RecoverPoint Release 3.1 Installation Guide


http://oss.oracle.com/projects/ocfs2/dist/documentation/v1.4/ocfs2-1_4-usersguide.pdf

http://oss.oracle.com/projects/ocfs2/documentation/v1.2/

http://e-generation.beasys.com/wls/docs92/admin.html

http://www.oracle.com/technology/tech/soa/index.html

http://download.oracle.com/docs/cd/B19306_01/server.102/b14239/toc.htm

http://www.emc.com/resource-library/resource-library.esp?source=link&findResourceBy=Products&searchKey=&listName=Products&grpName=Products&name=RecoverPoint

http://www.emc.com/collateral/software/white-papers/h2781-emc-recoverpoint-3-new-features.pdf

http://www.emc.com/collateral/software/white-papers/h4175-recoverpoint-concurrent-local-remote-oper-disaster-recovery-wp.pdf

http://www.emc.com/collateral/software/white-papers/h4175-recoverpoint-concurrent-local-remote-oper-disaster-recovery-wp.pdf

http://www.emc.com/collateral/software/white-papers/h2346-recoverpoint-ov.pdf

Appendix

Oracle DR terminology This appendix defines the following Oracle disaster recovery terminology:

• Application Server host name: This paper differentiates between the terms Application Server host name and network host name. The Application Server host name is the host name that Oracle Application Server uses for the host when Oracle Application Server is configured on the host. During installation, the installer automatically retrieves the Application Server host name from the current host and stores it in the Oracle Application Server configuration metadata on disk. A host can have only one Application Server host name. See also network host name later in this section.

• Asymmetric topology: A disaster recovery configuration that is different across tiers on the production site and standby site. In an asymmetric topology, the standby site can use less hardware (for example, the production site could include four hosts with four Application Server instances while the standby site includes two hosts with four Application Server instances. Or, in a different asymmetric topology, the standby site can use fewer Application Server instances. For example, the production site could include four Application Server instances while the standby site includes two Application Server instances). Another asymmetric topology might include a different configuration for a database (for example, using a Real Application Clusters database at the production site and a single instance database at the standby site).

• Disaster recovery: The ability to safeguard against natural or unplanned outages at a production site by having a recovery strategy for applications and data to a geographically separate standby site.

• Network host name: A host name assigned to an IP address that is resolved through DNS resolution. The network host name is the host name by which a particular host is known within the host's network. A host can have the same network host name and Application Server host name. A host can have only one Application Server host name, but it can have multiple network host names. See also Application Server host name earlier in this section.

• Production site setup: The process of creating the production site. To create the production site using the procedure described in this manual, you must plan and create Application Server host names and network host names, create mount points and links on the hosts to the Oracle home directories on the shared storage where the Oracle Application Server instances will be installed, install the binaries and instances, and deploy the applications.

• Site failover: The process of making the current standby site the new production site after the production site becomes unexpectedly unavailable (for example, due to a disaster at the production site). This paper also uses the term "failover" to refer to a site failover.

• Site switchover: The process of reversing the roles of the production site and standby site. Switchovers are planned operations done for periodic validation or to perform planned maintenance on the current production site. During a switchover, the current standby site becomes the new production site, and the current production site becomes the new standby site. This paper also uses the term "switchover" to refer to a site switchover.

• Site synchronization: The process of applying changes made to the production site at the standby site. For example, when a new application is deployed at the production site, you should perform synchronization so that the same application will be deployed at the standby site, also.

• Standby site setup: The process of creating the standby site. To create the standby site using the procedure described in this paper, you must plan and create Application Server host names and network host names, perform a switchover operation (which replicates the Oracle home directories and installations from the production site shared storage to the standby site shared storage), and create mount points and links to the Oracle home directories on the standby shared storage.


• Symmetric topology: An Oracle Application Server Disaster Recovery configuration that is

completely identical across tiers on the production site and standby site. In a symmetric topology, the production site and standby site have the identical number of hosts, load balancers, instances, and applications. The same ports are used for both sites. The systems are configured identically and the applications access the same data. This paper describes how to set up a symmetric Oracle Application Server disaster recovery topology for an enterprise configuration.

• Topology: The production site and standby site hardware and software components that comprise an Oracle Application Server disaster recovery solution.

RecoverPoint terminology • Bookmarks: A bookmark is a named snapshot. The bookmark uniquely identifies an image.

Bookmarks can be set and named manually; they can also be created automatically by the system either at regular intervals or in response to a system event. Bookmarked images are listed by name.

• Consistency group: A consistency group is a logical grouping of replication volumes that must be consistent across one another. The need for consistency across these volumes could be due to the volumes being used by the same application or needing to have the data on the volumes at the same point in time when recovered due to data dependencies. A consistency group is also used to determine replication direction and policies on a set of replication volumes. Each consistency group is an independent entity and can have different replication direction and policies than other consistency groups. This allows for synchronous and asynchronous replication as well as bi-directional replication to exist in the same environment. Consistency groups are a technology that groups together various objects, either on a single system or across systems, so that when they are moved or copied, they’re seen as a group. Remember, you either get all of the data or none of it—you don't want to get part of the data.

• Continuous data protection (CDP): Local replication across heterogeneous environments. At the local site, the CDP engine captures every I/O into the local CDP journal with I/O bookmarking to capture application events. CDP provides instantaneous or on-demand any-point-in-time recovery regardless of the array type.

• Continuous local and remote data protection (CLR): Simultaneous block-level local replication and asynchronous block level remote replication for LUNs with one copy residing locally in the same SAN, and the second copy residing remotely in a different SAN. Locally, every write is journaled. With remote replication significant groups of writes are journaled (bandwidth efficiencies). With CLR, users are enabled to independently recover from local or remote sites. Recovery of one copy locally or remotely can occur without affecting the other copy. This ability to fail over to a local copy of data without impacting a remote site extends RecoverPoint disaster recovery to encompass local as well as regional events.

• Continuous remote replication (CRR): Remote replication, with remote site recovery. It provides heterogeneous replication with policy-based bandwidth reduction and efficiencies in asynchronous or synchronous replication environments. CRR implements bi-directional, heterogeneous, block-level replication across any distance using asynchronous, synchronous, and snapshot modes.

• Journal: Provides time-stamped recovery points with application-consistent bookmarks. It also correlates system-wide events (port failure, system error, and so on) with potential corruption events, which is very useful when performing root-cause analysis. These application and system bookmarks are automatic, but users can also enter their own bookmarks into the system. The RecoverPoint journal, an important component of the protection process, provides the following capabilities: Tracks all data changes to every protected LUN. It saves each write so that it represents an any-

point-in-time image of a protected LUN. Utilizes bookmarks for application-aware recovery. Maintains all of the application, user, and environmental bookmarks associated with specific point-

in-time images.


Repository for live data updates. Maintains a reserved space, called the target-side processing space, which is used to store changes to an image that has been recovered.

Provisioned from existing SAN LUNs. Can be configured from any SAN-accessible LUN, or from a collection of “concatenated” LUNs. The size of the journal can be dynamically increased by adding a new LUN without discarding the existing history contents.

Dynamically compressed, which saves storage. The data is stored in a compressed format that can be used to roll a protected LUN back to any point in time.

• RecoverPoint appliance (RPA): Based on a standard Dell 1μ server running a customized Linux

kernel. The appliance has four 4 Gb/s Fibre Channel ports that are used to attach into a single- or dual-node (A/B) fabric. Each RPA also has two Ethernet ports—one is used as the management-control network and one is used to communicate to a remote RecoverPoint appliance cluster. It is designed for high availability with redundant power and cooling. The RecoverPoint application software provides core functionality and management for the system. Appliances are deployed in a two- to eight-node cluster configuration that allows active-active failover between the nodes. The RecoverPoint software is designed to avoid the split-brain issues that can arise with traditional clustering technologies. All RecoverPoint appliances are in constant communication and use a shared private SAN volume to maintain metadata state. If a RecoverPoint appliance fails, one of the other RecoverPoint appliances will take over without interrupting any in-progress CDP or CRR operations.

• Replication volumes: Volumes with data to be replicated. If the source and target replication volumes differ in size, then the source must be the smaller of the two volumes. This is typical in heterogeneous storage environments as well as some environments where different versions of storage management software are used. Any excess size will not be replicated and will be hidden from the host servers.

• Replication set: The association created between the source volume and the local and/or remote target volumes is called the replication set. A consistency group contains one or more replication sets.

• Repository volume: This volume holds the configuration and marking information during the replication. At least one repository volume is required per site and is accessible from all RPAs at the site.

• Snapshot: A snapshot is the difference between one consistent image of stored data and the next. Snapshots are taken seconds apart. The application writes to storage; at the same time, the splitter provides a second copy of the writes to the RecoverPoint appliance. In asynchronous replication, the appliance gathers several writes into a single snapshot. The exact time for closing the snapshot is determined dynamically depending on replication policies and the journal of the consistency group. In synchronous replication, each write is a snapshot. When the snapshot is distributed to a replica, it is stored in the journal volume, so that is it possible to revert to previous images by using the stored snapshots.

RecoverPoint write splitters The function of the splitter is to mirror writes from the application server to LUNs being protected by RecoverPoint. When a write is requested from the application server it is split and sent to the RecoverPoint appliance in one of three ways.

• The first method utilizes a host splitter/driver. This host splitter is a lightweight driver that resides in the I/O stack, below any file system and volume manager, and just above any multipath driver (such as EMC PowerPath). The splitter looks at the destination for the write packet. If the write is to a LUN that RecoverPoint protects, the splitter will send a copy of the write packet to the RecoverPoint appliance. It does this by rewriting the target address inside the packet to redirect it to the RecoverPoint appliance’s pseudo-LUN, and reissuing the write down the stack.

• The second method is through an intelligent fabric switch, either the Brocade Connectrix® AP-7600B or Connectrix ED-48000B with the SAS APIs, or one of the Cisco Connectrix MDS-9000 series


switches with the SANTap API. The switch intercepts all writes to LUNs being protected by RecoverPoint, and sends a copy of that write to the RecoverPoint appliance.

• The third method is through a CLARiiON array-based splitter, which is supported on CLARiiON CX3 arrays with FLARE 26 patch code, and CLARiiON CX4 arrays with FLARE 28 patch code. The array intercepts all writes to the LUNs being protected by RecoverPoint, and sends a copy to the RecoverPoint appliance.

In all cases, the original write travels though its normal path to the production LUN.

When the copy of the write is received by the RecoverPoint appliance, it is acknowledged back (ACK). This ACK is received by the splitter (the host, CLARiiON or fabric splitter), and held until the ACK is received back from the production LUN. With both ACKs received, the ACK is sent back to the host, and I/O continues normally.

Once the appliance has acknowledged the write, it will move the data into the local journal volume, along with a timestamp and any application-, event-, or user-generated bookmarks for the write. When the data is safely in the journal, it is then distributed to the target replica volumes, with care taken to ensure that write order is preserved during this distribution.

RecoverPoint image access modes To test and verify the replica at the disaster recovery site is a reliable and consistent image of the production site, it is necessary to access the image, as referenced in the procedures in this paper.

Image access is required to restore production from the disaster recovery site, and to roll back to a previous state of the data. It is also required to temporarily operate systems from a replicated copy while maintenance work is carried out on the production site and to fail over to the replica. When image access is enabled, host applications at the copy site can access the replica.

The following is a discussion of several of the available image access strategies that were pertinent in this implementation, that is, Logged, Virtual, Virtual with Roll, and Disable Image access.

Virtual access (instant) In Virtual access, the system creates the image selected in a separate virtual LUN within the RecoverPoint appliance. Performance is constrained by the RecoverPoint appliance; however access to the point-in-time image is nearly instantaneous. The image can be used in the same way as logged access (physical), but again, all data changes are temporary and are stored in a special place on the local journal.

Generally, this type of image access is chosen because the user is not sure which image, or point in time is needed. The user must access several images to conduct forensics and determine which replica is required.

As stated above, the image is accessed through the RecoverPoint appliance; virtual access is not recommended for heavy workloads or production work.

You will not be able to recover the production site from a virtual image. By definition, the image is temporary. Generally when work is completed, the choice is made to disable image access.

If it is determined the image should be maintained, then access must be changed to Logged access using Roll To Image. This can be done on the fly using a pull-down menu in the GUI or through the RecoverPoint command line interface. When you disable image access, the virtual LUN and all writes to it are discarded.

Virtual access (instant) with Roll image in background In Virtual access with Roll image in background, the system first creates the image in a virtual volume managed by the RecoverPoint appliance. This provides very fast access to the image, the same as in Virtual access. Simultaneously in background, the system rolls to the physical image. Once the system has completed this action, the virtual volume is discarded, and the physical volume takes its place. At this point, the system continues to function as if you had chosen Logged image access initially.


The virtual volume and the physical volume have the same SCSI ID. The virtual LUNs owned by the RecoverPoint appliance look like physical target LUNs but were dynamically created by the RPA. If you execute a SCSI inquiry against the virtual LUNs, the data returned will be the same as if the inquiry was invoked against the physical LUNs. Zoning and all other characteristics will appear to have the same configuration. The virtual LUNs can be mounted to the disaster recovery server and are seen by the operating system as physical LUNs.

The switch from virtual to physical will be transparent to the servers and applications. The user will not see any difference in access. Once this occurs, changes are read from the physical volume instead of being performed by the RecoverPoint appliance.

If you disable image access, the writes to the volume while image access was enabled will be rolled back (undone). Then distribution to storage will continue from the accessed image forward.

This type of access is recommended when the decision is made to move from the current production site to the disaster recovery site, that is, catastrophe, and immediate access to the image is required. If the intention is to roll back in time, but you need immediate access to images to determine which image is valid, this is also the best option. This option is also viable for a heavy workload. Lastly, as mentioned, production cannot be recovered from a virtual only image. This type of access or logged access is necessary.

Logged access (physical) In Logged access, the system rolls backward (or forward) to the snapshot (point in time) you select to access. There will be a delay while the successive snapshots are applied to the replica image to create the image you selected. The length of delay depends on how far the selected snapshot is from the snapshot currently being distributed to storage.

Once the access is enabled, hosts will have direct access to the replica volumes, and the RPA will not have access; that is, distribution of snapshots from the journal to storage will be paused.

When you disable image access, the writes to the volume while image access was enabled will be rolled back (undone). Then distribution to storage will continue from the accessed snapshot forward.

Logged access is the preferred image access for production. When recovering production from the disaster recovery site this image should be enabled.

Disable Image Access Choosing to disable image access means all changes to the replica will be discarded or thrown away. It does not matter what type of access was initiated, that is, logged or another type, or whether the image chosen was the latest or an image back in time.

When the splitter is disabled the LUN will be masked off. The operating system will still see the LUNs as mounted. But in fact, if you try to access data or rescan the disks using a SCSI command, the servers will report errors. Disabling the image is like yanking out the LUNs from underneath the application. Applications usually issue errors when data is cached or information is pending to be flushed from the operating system.

The cleanest way to ensure there will be no errors on the disaster recovery site is to first shut down the applications. Then unmount the file systems, and disable the image. Disabling image access restores the storage state to No access. Changes to the replica recorded in the image access log are automatically undone, so that the replica is restored to the state it was in before it was accessed. Using Disable Image Access effectively says the work done at the disaster recovery site is no longer needed. Some reasons may be that the point-in-time image chosen was not the correct image, or the information sought was obtained and propagated by another means.


disaster recovery of oracle fusion middleware and oracle ... · pdf filedisaster recovery of...

Documents