integration of ibm infrastructure and sap lvm v2.0

IBM SAP International Competence Center

SAP NetWeaver Landscape Virtualization Management on IBM Infrastructure © Copyright IBM Corporation, 2013 of 156

IBM SAP Technical Brief

Implementing and Using

SAP NetWeaver Landscape Virtualization Management

on IBM Power Systems and IBM Flex System POWER Nodes


Walldorf, Germany

Version: 2.0

Status: December 2013



Preface Edition Notice (December 2013)

SAP NetWeaver Landscape Virtualization Management software (SAP LVM) is the

successor and extension of the SAP Adaptive Computing Controller. This document is

complement to the ISICC Implementation Guide “SAP’s Adaptive Computing on Power

Systems”, Version 3.0, August 2011. The paper focuses on the new features and uses-cases

provided with SAP LVM in conjunction with IBM Systems Director managed Power

Systems. It describes the IBM infrastructure specifics required to implement and operate an

on-premise Cloud Solution for SAP landscapes.

The paper covers a solution stack tested during a Proof-of-Concept at the ISICC. This

includes the following components:

SAP Business Suite 7 with IBM DB2 UDB LUW 9.7/10.1 databases

IBM PureFlex Systems

AIX 6 and AIX 7 operating systems

IBM Flex System Manager

IBM Tivoli Storage FlashCopy Manager

IBM SAN Volume Controller

IBM XIV Storage System

At the time of writing, SAP LVM V2.0 SP3 is generally available.

Scope and Audience

People reading this document should understand core components and pre-requisites of the

SAP and Systems Software solution stack in an IBM POWER environment including their

installation and handling. They should understand the typical use-cases and their execution

and effects on a POWER/AIX based SAP infrastructure.

The document is intended for IBM (practitioners from Pre- & Post-Sales and Services,

GBS/GTS, ATS, eTS,) and customer personnel involved in the implementation of SAP

Landscape Virtualization Management software on IBM Power Systems. Since we do not

repeat existing documentation and technical information from both IBM and SAP the readers

are expected to have a solid knowledge in AIX, PowerVM virtualization, IBM Systems

Director and its plug-ins, IBM Flex System Manager, IBM Tivoli Storage FlashCopy

Manager and SAP Basis administration.

Authors

Dr. Edmund Haefele, IBM Technical Sales for SAP

Dr. Thomas Hebert, IBM Technical Sales for SAP

Maik Gasterstaedt, IBM SAP International Competence Center

Walter Orb, IBM SAP International Competence Center

Matthias Koechl, IBM SAP International Competence Center

Arnold Beilmann, IBM Research & Development

Christoph Langer, IBM Research & Development

Feedback

We are interested in any feedback you have. Please send your comments to

[email protected].

mailto:[email protected]



Disclaimer

This document is subject to change without notification and will not comprehensively cover

the issues encountered in every customer situation. It should be used only in conjunction with

the product literature accompanying the products listed above. The information contained in

this document has not been submitted to any formal IBM test and is distributed AS IS.



1 INTRODUCTION ....................................................................................................................................... 9

1.1 BACKGROUND AND BUSINESS CONTEXT ................................................................................................... 9 1.2 ENABLING END-TO-END MANAGEMENT THROUGH INTEGRATION ............................................................ 9 1.3 SCOPE OF THIS COOKBOOK ...................................................................................................................... 10 1.4 SAP NETWEAVER LANDSCAPE VIRTUALIZATION MANAGEMENT SOFTWARE (SAP LVM) .................... 11 1.5 SAP LVM VERSION 2.0 UPDATE............................................................................................................. 13

2 OVERVIEW AND PLANNING ............................................................................................................... 15

2.1 ARCHITECTURE OVERVIEW ...................................................................................................................... 15 2.2 COMBINING THE IBM AND SAP LAYERS ................................................................................................. 19 2.3 PLANNING PROJECT RESOURCES AND TIMING ......................................................................................... 25 2.4 TESTED CONFIGURATION ......................................................................................................................... 27 2.5 SAP LVM ENVIRONMENT ....................................................................................................................... 28

3 INSTALLATION ...................................................................................................................................... 37

3.1 NAMING CONVENTIONS ........................................................................................................................... 38 3.2 INSTALL STORAGE INFRASTRUCTURE ...................................................................................................... 39 3.3 INSTALL MANAGEMENT ENVIRONMENT .................................................................................................. 49 3.4 INSTALL SYSTEM LANDSCAPE ................................................................................................................. 57

4 INITIAL LVM CONFIGURATION ....................................................................................................... 78

4.1 ENGINE SETTINGS .................................................................................................................................... 78 4.2 VIRTUALIZATION MANAGER CONFIGURATION IN SAP LVM .................................................................. 78 4.3 STORAGE MANAGER CONFIGURATION .................................................................................................... 80 4.4 CONFIGURE SAP LVM POOLS ................................................................................................................ 82 4.5 CONFIGURE NETWORKS .......................................................................................................................... 83 4.6 DISCOVERY OF HOSTS (LPARS) .............................................................................................................. 84 4.7 INSTANCE CONFIGURATION/ DISCOVERY IN SAP LVM .......................................................................... 86 4.8 ADDITIONAL CONFIGURATION STEPS ...................................................................................................... 93 4.9 IBM FLEX SYSTEM MANAGER CONFIGURATION ..................................................................................... 94

5 DETAILED DESCRIPTION / EXECUTION OF USE CASES ........................................................... 99

5.1 LVM STANDARD EDITION USE CASES .................................................................................................... 99 5.2 SAP LVM ENTERPRISE EDITION USE CASES ........................................................................................ 109

6 TROUBLESHOOTING HINTS AND TRICKS ................................................................................... 151

6.1 MANUAL CLEANUP OF SAP LVM COPY PROCESS .................................................................................. 151 6.2 DELETE THE SYSTEM IN SAP LVM ....................................................................................................... 153

7 RESOURCES ........................................................................................................................................... 154

7.1 IBM DOCUMENTATION ......................................................................................................................... 154 7.2 SAP DOCUMENTATION AND SAP NOTES .............................................................................................. 154



List of figures

Figure 1 Logical connections of IBM and SAP components ................................................... 10

Figure 2 Generic SAP NetWeaver LVM capabilities and LVM 2.0 updates (green) .............. 11 Figure 3 SAP LVM Block Diagram (Source: SAP) ................................................................ 16 Figure 4 IBM Systems Director and Plug-Ins .......................................................................... 18 Figure 5 Software structure to integrate IBM IT layers ........................................................... 20 Figure 6 Generic SAP System Copy Workflow ....................................................................... 21

Figure 7 Architecture overview diagram ................................................................................. 28 Figure 8 Redundant virtual fibre channel attachment .............................................................. 33 Figure 9 System configuration in SAP LVM with distributed file system layout ................... 39 Figure 10 Example configuration of an SAP system using GPFS ........................................... 44

Figure 11 Volume Group and Filesystem layout ..................................................................... 46 Figure 12 Volumes of SAP system CL6 .................................................................................. 48 Figure 13 SAP System CL6 in LVM and mount points .......................................................... 49 Figure 14 Overview of the FlashCopy Manager Installation ................................................... 51

Figure 15 Sample LDAP Directory Information Tree for AIX user management .................. 54 Figure 16 Start IPsec ................................................................................................................ 60 Figure 17 Define the Host as “Isolation Ready” in SAP LVM ................................................ 60 Figure 18 Filesystem for Oracle Redo Logs ............................................................................ 72

Figure 19 LVM Engine Settings .............................................................................................. 78 Figure 20 Virtualization Manager Configuration ..................................................................... 78

Figure 21 Virtualization Manager: Virtualization Manager Types .......................................... 79 Figure 22 Virtualization Manager: Virtualization Manager Properties ................................... 79

Figure 23 Virtualization Manager: Summary .......................................................................... 80 Figure 24 Storage Manager Configuration ............................................................................... 80 Figure 25 Storage Manager: Storage Manager type ................................................................. 80

Figure 26 Storage Manager: Storage Manager Properties ....................................................... 81 Figure 27 Storage Manager: Storage Systems ......................................................................... 82

Figure 28 SAP LVM Pools: Pool Overview ............................................................................ 83 Figure 29 SAP LVM Network Configuration .......................................................................... 83 Figure 30 SAP LVM Network: Basic Configuration ............................................................... 84 Figure 31 SAP LVM Network: Advanced Configuration ....................................................... 84

Figure 32 SAP LVM Network: Summary ................................................................................ 84 Figure 33 SAP LVM hosts: Discover Using Host and Instance Agent ................................... 85 Figure 34 SAP LVM Hosts: Add Hosts ................................................................................... 85

Figure 35 SAP LVM Hosts: Edit host properties ..................................................................... 86 Figure 36 SAP LVM Systems: Discover new SAP instances .................................................. 86 Figure 37 SAP LVM Systems: Find Instances/Hosts .............................................................. 87 Figure 38 SAP LVM Systems: Add Instances – Modify Host Names .................................... 88

Figure 39 SAP LVM Systems: Add Instances – Auto Assign to New or Existing System ..... 88 Figure 40 SAP LVM Systems: Basic Configuration ............................................................... 89 Figure 41 SAP LVM Systems: Provisioning & RFC ............................................................... 89 Figure 42 SAP LVM Systems: Network Isolation ................................................................... 90 Figure 43 SAP LVM Systems: Network Isolation - Add allowed communications ............... 90

Figure 44 Database Configuration: Instance Details ................................................................ 91 Figure 45 Database Configuration: Basic configuration .......................................................... 91

Figure 46 Database Configuration: Instance properties ........................................................... 92 Figure 47 Database Configuration: Mount points .................................................................... 92 Figure 48 Central Services: Mount points ................................................................................ 93 Figure 49 Overview of systems and instances ......................................................................... 93



Figure 50 System and AS Provisioning Configuration ............................................................ 94 Figure 51 System Copy Configuration: Release configuration type ........................................ 94 Figure 52 System Copy Configuration: Basic properties ......................................................... 94 Figure 53 System Copy Configuration: Installation master ..................................................... 94

Figure 54 IBM Flex System Manager Security: Group membership ...................................... 96 Figure 55 IBM Flex System Manager Security: Create new roles .......................................... 96 Figure 56 IBM Flex System Manager Security: Modify role permissions .............................. 97 Figure 57 IBM Flex System Manager Security: Define static groups ..................................... 98 Figure 58 IBM Flex System Manager Security: Assign groups and roles to user group ......... 98

Figure 59 IBM PowerVM Relocate ....................................................................................... 101 Figure 60 SAP LVM Relocate ............................................................................................... 101 Figure 61 Prepare SAP Instance ............................................................................................. 103

Figure 62 Start SAP system: Mass Operations ...................................................................... 104 Figure 63 Logs of a 'Start SAP system' Operation ................................................................. 104 Figure 64 Relocate SAP System: Mass Operation ................................................................. 105 Figure 65 Shutdown Host ....................................................................................................... 106

Figure 66 SAP LVM Dashboard ............................................................................................ 107 Figure 67 SAP LVM Infrastructure Visualization ................................................................. 108 Figure 68 SAP LVM Performance Monitoring ...................................................................... 109 Figure 69 SAP System Copy Scenarios Supported by SAP LVM (Source SAP) ................. 110

Figure 70 Enable SAP System for SAP System Rename ...................................................... 112 Figure 71 Start SAP System Clone ........................................................................................ 114 Figure 72 SAP System Clone: Basic Data of Target System ................................................. 114

Figure 73 SAP System Clone: Host Selection of Target System ........................................... 115

Figure 74 SAP System Clone: Virtual Host Names and Networks ....................................... 117 Figure 75 SAP System Clone: Storage Volumes ................................................................... 117 Figure 76 SAP System Clone: Database consistency ............................................................ 118

Figure 77 SAP System Clone: Define Allowed Outgoing Connections for System Isolation

................................................................................................................................................ 118

Figure 78 SAP System Clone: Summary ............................................................................... 119 Figure 79 Define RFC Destination for PCA Execution ......................................................... 121 Figure 80 Start SAP System Copy ......................................................................................... 121

Figure 81 SAP System Copy: Basic Data of Target System .................................................. 122 Figure 82 SAP System Copy: Host Selection of Target System ............................................ 122

Figure 83 SAP System Copy: Virtual Host Names and Networks ........................................ 123 Figure 84 SAP System Copy: SAP Instance Numbers .......................................................... 123

Figure 85 SAP System Copy: Storage Volumes .................................................................... 124 Figure 86 SAP System Copy: SVC CLI Warning Message .................................................. 124 Figure 87 SAP System Copy: Database consistency ............................................................. 125 Figure 88 SAP System Copy: User and Group Management ................................................ 125 Figure 89 SAP System Copy: SAPinst Release Configuration .............................................. 126

Figure 90 SAP System Copy: Define Allowed Outgoing Connections for System Isolation 126 Figure 91 SAP System Copy: Post Copy Automation ........................................................... 127 Figure 92 SAP System Copy: Select Task List ...................................................................... 127 Figure 93 SAP System Copy: Summary View ...................................................................... 128 Figure 94 SAP System Refresh: Start System Refresh .......................................................... 129

Figure 95 SAP System Refresh: Basic Data of System to be Refreshed ............................... 129 Figure 96 SAP System Refresh: Host Selection of Target System ........................................ 130

Figure 97 SAP System Refresh: Virtual Host Names and Networks .................................... 130 Figure 98 SAP System Refresh: Storage Volumes ................................................................ 131 Figure 99 SAP System Refresh: Database consistency ......................................................... 131



Figure 100 SAP System Refresh: User and Group Management .......................................... 132 Figure 101 SAP System Refresh: SAPinst Release Configuration ........................................ 132 Figure 102 SAP System Refresh: Define Allowed Outgoing Connection for System Isolation

................................................................................................................................................ 133

Figure 103 SAP System Refresh: Post Copy Automation ..................................................... 133 Figure 104 SAP System Refresh: Storage Volumes .............................................................. 134 Figure 105 SAP System Refresh: Summary .......................................................................... 135 Figure 106 Start SAP System Rename ................................................................................... 136 Figure 107 SAP System Rename: Basic Data of Target System ........................................... 136

Figure 108 SAP System Rename: Virtual Hostnames and Networks .................................... 137 Figure 109 SAP System Rename: SAP Instance numbers ..................................................... 137 Figure 110 SAP System Rename: Mount Data ...................................................................... 138

Figure 111 SAP System Rename: User and Group Management .......................................... 139 Figure 112 SAP System Rename: SAPinst Release Configuration ....................................... 139 Figure 113 SAP System Rename: Define Allowed Outgoing Connections for System Isolation

................................................................................................................................................ 140

Figure 114 SAP System Rename: Post Copy Automation .................................................... 140 Figure 115 SAP System Rename: Summary .......................................................................... 141 Figure 116 Provider Implementation Definition (I) ............................................................... 144 Figure 117 Provider Implementation Definition (II) .............................................................. 144

Figure 118 Provider Implementation Definition (III) ............................................................ 145 Figure 119 Provider Implementation Definition (IV) ............................................................ 145 Figure 120 Custom Hook Definition (I) ................................................................................. 145

Figure 121 Custom Hook Definition (II) ............................................................................... 146

Figure 122 Custom Hook Definition (III) .............................................................................. 146 Figure 123 System State in Operations View ........................................................................ 147 Figure 124 Monitoring View .................................................................................................. 148

Figure 125 Mass Stop and Unprepare Completed ................................................................. 148 Figure 126 Destroy System .................................................................................................... 149

Figure 127 System Destroy: Delete Storage Volumes ........................................................... 149 Figure 128 System Destroy: Host Names .............................................................................. 150 Figure 129 System Destroy: Summary .................................................................................. 150

Figure 130 Remove System in SAP LVM ............................................................................. 153



List of tables

Table 1 Core Functionalities in SAP LVM .............................................................................. 13

Table 2 Supported SAP releases for the SAP LVM scenario .................................................. 15 Table 3 Overview of the Scenarios .......................................................................................... 24 Table 4 Project Phases and estimated time .............................................................................. 27 Table 5 Software Stack ............................................................................................................ 28 Table 6 Operational Model ...................................................................................................... 37

Table 7 Sample directory structure for a distributed file system implementation ................... 39 Table 8 Entries in /etc/services ................................................................................................. 61 Table 9 Local Filesystem layout .............................................................................................. 62 Table 10 XIVGUI Install directory .......................................................................................... 63

Table 11 Minimum filesystem layout (DB2 UDB) .................................................................. 69 Table 12 Extended filesystem layout (DB2 UDB) ................................................................... 70 Table 13 Minimum filesystem layout (Oracle) ........................................................................ 70 Table 14 Extended filesystem layout (Oracle) ......................................................................... 71

Table 15 Storage Connection Methods .................................................................................... 81 Table 16 Storage Connection user credentials ......................................................................... 82 Table 17 Required Setup Procedures ..................................................................................... 113 Table 18 Examples for IP addresses ...................................................................................... 115

Table 19 List of SAP Notes .................................................................................................... 155



1 Introduction

1.1 Background and Business Context

In the era of worldwide business relations and processes, the flexible and non-disruptive

operation of business applications and their underlying middleware and hardware becomes an

essential factor for success. Even unforeseen fluctuations in transaction volumes and changing

business processes must be reflected immediately by the supporting system capacities. On the

other hand, planned interventions such as hardware or application maintenance and testing

must not disrupt SAP Business Suite availability to worldwide end users, partners and

consumers. In order to accelerate innovation and fast adoption of new business processes,

non-production stages of SAP systems need to be deployed on demand, without a long lead-

time for procurement and installation.

Consequently, manually operating and administrating a growing number of individual

systems or landscape components is no longer an option. The concepts of cloud computing

promise to save costs and will increase flexibility, elasticity, and automation of system

operations to efficiently serve the needs of the business.

This paper describes how to design and implement an on-premise cloud environment by

combining IBM platform technologies, namely IBM Flex System Manager, IBM Systems

Director, and Tivoli Flash Copy Manager with the SAP NetWeaver® Landscape

Virtualization Management (SAP LVM) environment. Finally, the paper shows how all the

use-cases, including cloning, copying, and refreshing of SAP systems, are managed via the

SAP LVM administrator user interface.

1.2 Enabling End-to-End Management through Integration

The major advantage of the solution stack described herein is the enablement of end-to-end

administration scenarios. This covers both aspects of IT management and SAP basis

administration by:

1. Providing a unified administrator interface for SAP topology monitoring and

launching automated tasks

The SAP basis administrator is familiar with SAP terminologies and user interfaces.

Hence, infrastructure complexity and platform specialties are masked and abstracted to

that level. Via the SAP LVM Administrator GUI IT tasks are provided to the SAP

administrator as self-services. At the same time, the IT administrator keeps control

over the infrastructure when enabling the self-services using IBM Systems Director or

IBM Flex System Manager.

2. Programmatic integration of IBM and SAP software layers for automation

SAP knows best how to manage SAP components and applications. IBM knows best

how to manage the infrastructure components consisting of IBM servers, storage,

virtualization technology and operating systems. For this reason, IBM platform

capabilities and software are used for all infrastructure related tasks. For the same

reason, SAP NetWeaver Landscape Virtualization Management complements these

for all management and automation functions related to the SAP software stack.



Figure 1 Logical connections of IBM and SAP components

The combination of IBM platforms’ native virtualization and automation capabilities with the

SAP management products establishes an excellent foundation for deploying and operating

SAP landscapes in an on-premise cloud environment. Consumers of the cloud services are the

administrators, not the end-users, though. Line of Business users benefit from faster

availability of systems, more flexible testing environments, and demand driven capacity of

server and storage resources.

1.3 Scope of this Cookbook

This cookbook describes the specifics and installation steps for implementing the SAP

NetWeaver Landscape Virtualization Management software V2.0 on an IBM PureFlex

System infrastructure. It also provides how-to’s for using the resulting administration

environment for launching and monitoring the most popular SAP “cloud use cases”.

The installation sequence has been tested in a Proof-of-Concept including

SAP Business Suite 7

DB2 UDB LUW

IBM POWER

AIX 6 and AIX 7



SAN attached disk storage using

o IBM System Storage SAN Volume Controller

o IBM Storwize V7000

o IBM XIV Storage System

Because of the technical affinity of Flex System Manager (FSM) and IBM Systems Director,

the content applies to standalone Power systems too. This has been verified in a lab

environment and in various customer proof of concept projects.



SAP LVM V1.0 became available in June 2012 and was baseline for the initial version of this

document. Since October 2013, SAP LVM version 2.0 is generally available. Our team

participated in the SAP LVM 2.0 ramp-up program since April 2013. Therefore, the current

document is based on SAP LVM 2.0 already.

The fact that we describe a POWER based implementation does in no way mean other

hardware and virtualization platforms (including IBM System x) do not offer a comparable

degree of SAP LVM integration.

The cookbook does not replace a solid technical knowledge of any of the involved layers and

is no substitute for existing literature by IBM and SAP. We will point to applicable literature

where needed.

1.4 SAP NetWeaver Landscape Virtualization Management software (SAP LVM)

SAP NetWeaver® Landscape Virtualization Management software allows customers to

monitor and manage entire SAP software landscapes and their associated heterogeneous IT

infrastructures. SAP LVM provides a single point of control, which allows visualizing,

monitoring, and managing data center tasks. The managing aspect includes resource

allocation and de-allocation on IT infrastructure level, but also on SAP Business Suite

application level.

Figure 2 Generic SAP NetWeaver LVM capabilities and LVM 2.0 updates (green)

SAP LVM is an evolution of the previously established SAP Adaptive Computing Controller

(ACC 7.3). As such SAP LVM inherits functions like “Landscape-wide Visualization and

Monitoring” and “Automated Capacity Management” from the former ACC.

In addition, SAP LVM provides new and more powerful functions like “SAP System

Cloning” and “SAP System Copy / Refresh”, which are frequently performed operations at

SAP sites. They combine numerous steps on infrastructure and SAP application level



including Post Copy Automation (PCA) sequences. SAP PCA is only available with the SAP

LVM “Enterprise” Edition.

SAP LVM Standard Edition is included in a SAP NetWeaver 7.x license without additional

fees. The SAP LVM Enterprise Edition is a separately licensed product with additional SAP

license fees.

SAP LVM provides all the administrative functionalities as listed in Table 1. Functions that

require SAP LVM Enterprise Edition are highlighted in blue and italic font.

The Standard Edition establishes the basic framework and includes administrative functions

that have been available with SAP Adaptive Compute Controller (SAP ACC) in the past.

These are related to landscape monitoring and automated startup, stop, and relocate of SAP

instances.

The Enterprise Edition extends these core capabilities with the advanced functions including/

involving SAP Post Copy Automation sequences. One can mostly automate complex and

frequently applied SAP system management tasks. This includes generation of isolated clones

of a SAP production system for testing purposes, but also replication of a production system

into a QA or pre-production-system that stay fully integrated within the existing transport

stages. For latter purpose, business data and system settings need to be replicated, while at all

times avoiding any impact on the online production environment.



SAP LVM Functionalities

End-to-End SAP System Clone / Copy / Refresh for ABAP

End-to-End SAP System Clone / Copy / Refresh for JAVA

Post-Copy Automation (PCA) for ABAP and JAVA

Automatic Capacity Management (ACM)

Application Server Installation / Un-installation

Mass Operations (start / stop / relocate)

User Configurable Dashboards

Reporting

Landscape Visualization

Custom Operations and Custom Hooks (Extensibility)

Custom Services (Management of Non-SAP Components)

Table 1 Core Functionalities in SAP LVM

1.5 SAP LVM Version 2.0 Update

SAP LVM version 2.0 introduced a number of new features. Evolutionary enhancements have

been made regarding architecture and functionality, and new features had been implemented.

The affected areas are highlighted in Figure 2. The most important changes are described in

the following three categories:

SAP LVM Administrator GUI Changes

The user interface allows for more customization. In addition, visualization of SAP

landscapes has been improved. Here it is important to note that SAP has changed terminology

of managed entities:

LVM 1.0 SAP 2.0

Service Instance or System

Resource Host

Controller Log Log

Customer scripts can extend the default SAP LVM functionality. For these custom operations,

new parameters can be defined and applied to managed systems.

A search facility allows you to easier identify and navigate to certain instances.

System Provisioning Changes

In general, LVM 2.0 has introduced some application aware functions for provisioning tasks.

E.g., the near-zero downtime (nZDM) cloning scenario for SAP NetWeaver Portal (in SP8)

provides a pre-canned procedure for the implementation of support package stacks on a

Portal system at minimum business downtime. This minimizes business downtime for all

users accessing SAP backend systems through this portal.

Further, end-to-end copy and refresh workflows have been extended respectively introduced

for SAP ERP/ECC, CRM, SCM and SAP NetWeaver BI. Optionally, these can include the

customizable post-copy automation (PCA) phase. PCA can also be launched as standalone

sequence on a target system provisioned by other means than SAP LVM.



When executing default LVM sequences enhanced custom hooks increase customization

capabilities for platform partners and customers. These hooks are either executed pre-, post,

or on-error of operations.

Platform Support and Integration Changed

Cluster awareness is an interesting point here. However, it does not mean that we now can

fully manage PowerHA clusters with LVM 2.0. Currently, this statement means detection and

monitoring of multi-node DBs like DB2 LUW DPF or Oracle RAC is supported. So

administrators are alerted when manipulating a DB-node. The same applies to shadow DBs,

which are mainly used for D/R purposes.

For programmatic use and integration into an enterprise monitoring and management system

extended outbound interfaces make LVM 2.0 an information provider to any of these 3rd

party

solutions. In addition, LVM can actively emit notifications about certain events or activities to

such solutions.

Changes Concerning Supported IBM Storage Solutions

With SAP LVM 2.0 the matrix of supported IBM storage solutions was extended. SAP LVM

2.0 supports all IBM storage solutions already supported in SAP LVM 1.0 (IBM XIV, IBM

SAN Volume Controller and IBM Storwize V7000) and introduces support for IBM System

Storage DS8000.

SAP LVM 2.0 improves the execution of SAP system clone, copy, and refresh operations. It

provides is more flexibility for specifying characteristics of storage volumes for SAP target

systems.

The IBM storage solutions now allow to

define names for AIX volume groups and logical volumes for the target systems

specify the storage pool in the storage system where the target storage volumes will be

allocated

create target SAP system on remote storage using a pre-configured storage mirroring

setup

The setup and configuration for SAP system clone, copy, refresh operations with IBM storage

solutions was simplified. No additional IBM specific scripts are required anymore. Support

for SMI-S communication with storage systems was added. Other communication methods,

used in SAP LVM 1.0 before, are now deprecated, but still supported.

Changes Concerning Supported IBM Virtualization Solutions

With SAP LVM 2.0 the integration with IBM Flex System Manager and IBM Systems

Director was improved. Server System Pools in IBM Systems Director VMControl are now

supported and may be used as a target for relocation or deployment of logical partitions. For

deployment of logical partitions, you may specify multiple network adapters and use IBM

Shared Storage Pools as image repository. You may capture logical partitions as image and

delete captured images directly from SAP LVM user interface.



2 Overview and Planning

2.1 Architecture overview

This section provides an introduction where and how individual functions are implemented.

2.1.1 SAP NetWeaver Landscape Virtualization Management

2.1.1.1 Technical Implementation

SAP LVM is an Java application that runs on add-on to SAP NetWeaver Application Server

Java. The required base version for SAP AS Java is 7.30, 7.31 or 7.40. Recommended

minimum support packages are SAP NetWeaver Release 7.30 Support Package 08, SAP

NetWeaver release 7.31 support package 04 or SAP NetWeaver release 7.4 support package

04.

The SAP LVM software is available for any platform that supports those SAP NetWeaver

releases.

SAP LVM can manage SAP systems running with older releases too. Table 2 lists the

supported releases for the managed SAP systems in a SAP LVM scenario. SAP note 1783702

describes in details the restrictions that may apply to some of the older SAP releases.

Supported Managed SAP systems (See SAP Note 1783702)

NetWeaver 7.40 NetWeaver 7.30 NetWeaver 7.20 NetWeaver 7.11 NetWeaver 7.10 NetWeaver 7.00 Web AS 6.40 Web AS 6.20 Web AS 6.10 R/3 4.6D R/3 4.6C

Table 2 Supported SAP releases for the SAP LVM scenario

Figure 3 illustrates the communication between all the different components. SAP LVM

software (both the SAP NetWeaver AS Java and the SAP LVM Add-on) is installed on a

single system designated as “Management Server. SAP LVM communicates with the

managed nodes via the SAP Host Agent (a process running on each managed node) for

landscape-wide monitoring and execution of tasks on the systems. Additionally there are

direct interfaces (black boxes in the diagram) to the IBM PowerVM virtualization layer and

the IBM Storage Layer. The IBM solution described here uses of all those paths.



Figure 3 SAP LVM Block Diagram (Source: SAP)

SAP LVM provides a set of APIs as interface. In order to support specific hardware,

virtualization, OS and DB layers, each SAP partner needs to supply a set of libraries that

translate the generic SAP LVM operations to platform specific commands and executes them

as integrated solution on the appropriate component. SAP Note 1783702 describes all the

supported platforms for SAP LVM 2.0. Not all platform providers have implemented specific

libraries for all of their platforms, or partly they support just a subset of the functionality.

Thus, the supported use case for the SAP LVM solution may vary by platform.

2.1.2 IBM Systems Management Components

In today’s complex and demanding environment, the IT infrastructure needs to react quickly

and adapt to the changing needs of the business. More and more of the IT infrastructure is

being virtualized and administrators must manage complex interactions between

virtualization, compute, storage, and network platforms. System administrators face

extraordinary demands as they try to plan, document, and roll out IT infrastructures, identify

capacity needs, get the most out of current assets, and contend with constant budget pressure.

IBM’s Systems Management Components free-up the IT administrators from repetitive, time-

consuming tasks allowing them to pursue higher value projects and accelerate innovation.

With more automation and integrated management across infrastructure elements, IT

administrators can focus a greater portion of their time on programs that drive innovation and

business advantage instead of daily routine tasks.

IBM Flex Systems Manager (FSM) and IBM Systems Director are platform management

tools. FSM builds upon IBM Systems Director code and its advanced plugins, but both were

enhanced with specific elements.

They have different specifications and support different types of hardware:

Flex Systems Manager (FSM) is a specialized tool for management of IBM Flex

Systems and IBM PureFlex Systems

IBM Systems Director is a general purpose systems management tool



2.1.2.1 IBM Systems Management Components for IBM PureFlex Systems

IBM Flex Systems Manager (IBM FSM) is a pre-installed appliance on a specialized compute

node for Flex and PureFlex systems. It is optimized for managing the Flex and PureFlex

systems. IBM FSM has a deep integration with hypervisors and system patterns for the

configuration of the environment on the Flex Chassis. The FSM Explorer user interface (UI)

of IBM FSM contains additional functionality and allows a graphical view to the components

in the Flex Chassis, the operating system provisioning engine and the configuration patterns

for Flex System components.

IBM FSM provides a single point of control for management of physical and virtual compute,

storage and networking resources from a single management console. IBM FSM enables fast

deployment operations of new partitions by simple and repeatable hardware configuration and

bare metal provisioning for server nodes, and five click creation and zoning of storage

volumes. IBM FSM allows the administrator to monitor the infrastructure from anywhere

using iOS, Android and Blackberry mobile devices. IBM FSM is designed to help to get the

most out of the IBM PureFlex System or IBM Flex System by automating repetitive tasks,

and providing visibility and control across compute, storage, network, and virtualization

functions within the data center.

IBM FSM management capabilities include:

Server management

IBM FSM auto-discovers hardware and manages inventory that supports up to 16

managed chassis, 224 compute nodes and 5,000 managed elements. IBM FSM

monitors hardware’s overall health status to proactively resolve issues. IBM FSM

detects and recovers potential problems in the PureFlex environment through an event

setup that triggers alerts and actions.

Virtualization management (VMControl Express)

o Create, edit, manage, and relocates virtual machines for rapid deployment

o Discover virtual machines, storage and network resources and visualize the

physical-to-virtual relationships

Advanced virtualization management (VMControl Standard and Enterprise):

o Automate the virtualized environment with system pools

Storage management (Storage Control)

o Storage provisioning for image creation, deployment, and cloning

o Shows relationships between storage and server resources

o Policy based storage placement and provisioning.

Fabric management (IBM Fabric Manager)

o Simplify and manage assignments of Ethernet MAC and Fibre Channel WWN

addresses using an enhanced UI

o Enables quick and easy deployment, configuration and recovery of compute

nodesI

o Monitors the health of compute nodes, and automatically without user

intervention replaces a failed compute node from a designated pool of spare

compute nodes

Network management (Network Control)

Network resource virtualization and graphical topology view of network resources and



connectivity end to end in a single tool.

Energy management (Active Energy Manager)

This feature is activated and monitors chassis by default. With simple real-time power

and thermal management across compute node, storage, networking and facility

providers it can set thresholds on hardware components to drive lower energy costs.

Security management

Provide administrative capabilities, such as setting up users within profile groups,

assigning security level of policies and security governance. FSM can view the access

state of managed resources in a web interface display.

The integration with SAP LVM leverages the virtualization management of

VMControl and Storage Control.

2.1.2.2 IBM Systems Management Components for IBM Power Systems

IBM Systems Director® provides similar functionality in an IBM Power Systems

environment compared to IBM FSM in an IBM PureFlex System landscape. In contrast to

IBM FSM (which is delivered as an appliance), IBM Systems Director must be manually

installed on a physical or virtual server.

IBM Systems Director is the platform management backbone, providing all the building

blocks for integrated services management. It provides the IT administrator with the features:

Unified management of physical and virtual resources for IBM servers, storage, and

networks.

Automated data center operations by implementing cloud-ready virtual infrastructures.

Figure 4 IBM Systems Director and Plug-Ins



Key value of IBM Systems Director is its ability to work in diverse IT environments, so it

reduces the number of required management tools and interfaces. IBM Systems Director can

manage rack, tower, and blade servers across all IBM brands. IBM Systems Director can

manage Flex Systems also, but only in case that no FSM is installed in the Flex Chassis.

Optional plug-ins can extend IBM Systems Director. The SAP LVM integration exploits:

IBM® Systems Director VMControl™ is a multi-platform virtualization management

solution. This solution – available as a plug-in for IBM Flex System Manager or for IBM

Systems Director – provides the following features that are also relevant in any SAP

landscape:

Create and manage virtual machines

Relocate virtual machines

Import, edit, create and delete virtual images

Deploy new virtual machines from virtual images

The VMControl plug-in is used by SAP LVM to perform the partition and AIX provisioning

steps along the overall processing sequence.

IBM® Systems Director Storage Control is another plug-in for IBM Flex System Manager

or for IBM Systems Director. It extends the management of systems to include the storage

layer. It is based on IBM Tivoli Storage Productivity and enables functions as:

Extends storage management of IBM Systems Director to cover most IBM storage

systems

Storage device discovery and coverage in integrated physical and logical topology views

Show relationships between storage and server resources

Ability to configure logical and physical configuration

Ability to view controller and volume status and to set notification alerts

Integration with IBM Systems Director VMControl storage provisioning for image

creation, deployment, and cloning

Storage Control is used in the end-to-end provisioning sequence to allow “bare-metal”

LPAR creation and AIX installation: During the provisioning process, Storage Control

takes care that the storage for the OS is provisioned and attached to the LPAR.

2.1.2.3 Storage Management (for Snapshots/FlashCopies)

IBM Tivoli® Storage FlashCopy® Manager. This software provides fast application-aware

backups and restores by exploiting snapshot technologies in IBM storage systems.

In the context of SAP landscape management, IBM Tivoli FlashCopy Manager is used to

create instantaneously consistent copies of online SAP systems. Instead of using these copies

as source for backups, SAP LVM uses the flashes as source for further processing system

replica by subsequent post-processing steps.

2.2 Combining the IBM and SAP Layers

2.2.1 SAP LVM Interfaces to Management Components



Figure 3 shows how the SAP LVM architecture is implemented for IBM POWER and IBM

storage platforms. The integration consists of a set of libraries, which transfer generic SAP

LVM commands into the specific syntax for the IBM environment. IBM provides these

libraries as part of the SAP LVM software stack.

Different communications paths exist between the managing server and the managed nodes:

Communication path via the SAP Host Agent

The SAP Host agent can execute OS commands on each managed SAP node. SAP

LVM communicates with this agent, which calls the vendor libraries in order to map

the LVM commands to an OS, DB, or storage specific language and syntax.

Communication path using the SAP LVM virtualization adapter

The managing system interfaces directly with the IBM Power HMC or with the

VMControl plug-in of IBM Flex System Manager/IBM Systems Director to control

PowerVM and OS layers.

Communication path using the SAP LVM storage adapter (IBM Storage Adapter)

For non-disruptive copies of running SAP instances, SAP LVM interacts with the

Tivoli Storage Flash Copy Manager, which interfaces to the storage system for the

cloning, and handles then handles all storage activities required on the managed nodes

also. This implementation guarantees a consistent replication of SAP instances by

synchronizing the required steps among the involved infrastructure layers (storage,

AIX, DB).

Figure 5 Software structure to integrate IBM IT layers



2.2.2 Example Workflow “SAP System Copy”

In this section, a walk-through through a sample “SAP System Copy” use case will

demonstrate how the integration works together. The workflow is composed of two phases

(See Figure 6):

System Cloning Phase

The system cloning phase begins by creating host name mappings on a name server

and is finished when the clones SAP systems is ready to be started in an isolated

network segment. These steps are the same for the system copy and system cloning

scenarios. At the end of this phase, a 1:1 clone of the source SAP system exists which

per default cannot be accessed from other system systems from outside that fenced

segment.

System Copy Phase

After the system-cloning phase is completed, SAP LVM triggers additional automated

tasks that make up the system copy phase. This includes renaming the SAP system

clone. The Post Copy Automation (PCA-) sequence adjusts the complete SAP

configuration so that finally a unique SAP system (SID) is established.

Figure 6 Generic SAP System Copy Workflow

We assumes the following landscape example for a detailed description of the workflow:

The source system is an SAP application server ABAP system

It is deployed in an AIX LPAR

The SAP system is configured with virtual IP addresses

The operating system, SAP system, and database are installed on separate AIX volume

groups

The physical disks for these volume groups are allocated on SAN storage systems.

The source system is configured in SAP NetWeaver Landscape Virtualization

Management

The administrator navigates to the “Provision view” in the SAP LVM console and selects the

source SAP system (All the detailed steps are shown later in chapter 5.2). After pressing the

Copy button, SAP LVM guides the administrator through dialogue steps defining the end-to-

end process. SAP LVM knows all the configuration settings of the registered source system;

however, the administrator has to specify configuration parameters for the target system.



Reasonable values for some parameters are determined automatically and pre-set as default

during the configuration dialogue. Following parameters can be changed (this is a non-

exhaustive list):

Target host: SAP NetWeaver Landscape Virtualization Management provides

the option to choose an already existing AIX LPAR or to create a new AIX

partition as target for the system. (For the latter the virtualization manager

configured in SAP NetWeaver Landscape Virtualization Management needs to

be set-up for provisioning and configuring such an operating system partition).

Host names: The copied SAP system is setup with a new virtual IP address

(and IP label) for each of the SAP instances. SAP NetWeaver Landscape

Virtualization Management is able to create new addresses and update the

domain name server automatically, otherwise the addresses for the target SID

need to be pre-configured in DNS.

Storage volumes: SAP NetWeaver Landscape Virtualization Management

uses the registered storage adapter to gather the data about the volumes

attached to the source system. It collects this information at runtime directly

from the storage management system. The administrator can specify target

volume names and change mount points, if required.

Database consistency: SAP NetWeaver Landscape Virtualization

Management asks whether the SAP source system is to be stopped during the

cloning process or whether the system should stay up and running (for

example, in case of a production system). With the online mode, the

administrator decides whether SAP NetWeaver Landscape Virtualization

Management is responsible for database consistency. Otherwise, there is no

guarantee that the cloned database can be recovered and brought online. When

selecting database consistency, SAP NetWeaver Landscape Virtualization

Management switches the source database into online backup mode (Oracle) or

into write suspend I/O mode (DB2). This ensures a consistent database image

during the actual cloning step.

Target isolation: On the target system network, network fencing has to be

established to ensure that the cloned system can be started without interfering

with other systems in the data center. This is achieved by using the IPsec

feature of AIX. SAP NetWeaver Landscape Virtualization Management allows

the definition of permitted outgoing network connections. All other outbound

connections are blocked while incoming connections are permitted.

SAP copy parameters: SAP NetWeaver Landscape Virtualization

Management queries information about the copied SAP system, such as the

new SAP system identifier (SAP SID) and master password.

SAP copy users: SAP NetWeaver Landscape Virtualization Management

allows new users to be created and defined for the copied SAP system on a

central user repository.



SAP post copy automation: SAP NetWeaver Landscape Virtualization

Management allows the selection of task lists and task list variants to be used

for post copy automation. This is possible for multiple clients in the SAP

system (such as basis tasks in client „000‟ and BDLS in client „100‟).

After all the configuration parameters have been defined, the administrator reviews the input

and then starts the SAP system copy workflow.

There is a narrated ScreenCam video available on TechDocs, in case you want get a real

impression about the look and feel of the SAP LVM administrator console.

2.2.3 Supported Use-Cases and involved components

The use-cases for

Landscape-wide overview and system monitoring incl. virtualization layers (physical

servers and LPARs)

Activation, deactivation, and relocation of SAP LVM hosts (LPARs)

Provision of new SAP LVM hosts (LPARs)

are included in the standard edition of SAP LVM and will leverage one of the virtualization

adapters during execution. These use-cases are not dependent on special storage, as long as

the storage satisfies the requirements of the virtual environment.

IBM Tivoli Storage FlashCopy Manager is not required for execution of these use-cases.

Enhanced functionalities (e.g. customized dashboards, generation of reports) will require the

enterprise edition.

The use-cases for

Attaching, detaching, relocating of SAP systems

Cloning, copying, refresh of SAP systems

have some dependencies on the storage infrastructure. SAP LVM manages the storage

volumes belonging to SAP systems during the execution of these operations.

The implementation of attach, detach, relocate operations of SAP systems depends on the

actual storage type.

The functionality for NAS and GPFS is included in the IBM storage library. For SAN-based

IBM storage, the IBM Storage Adapter is also required in addition to the IBM storage library.

Clone, copy, and refresh operations require SAN based IBM storage, IBM Storage Adapter,

and IBM storage and platform libraries.

Integration with other, non-IBM storage (like EMC or NetApp) is possible but not covered in

this paper.

Table 3 summarizes the different use case families in SAP LVM and illustrates which Edition

for SAP LVM is required, which SAP LVM adapters needs to be invoked, and which

boundary conditions need to be fulfilled for the execution.

http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4949



Use Case Landscape

Overview System monitoring

VM (LPAR) Activate Deactivate Relocate

VM (LPAR) Provision new LPARs

SAP System Attach Detach Relocate

SAP System Clone Copy Refresh

SAP LVM Adapter

Virtualization Adapter

Not required for NFS and GPFS, Storage Adapter for SAN based storage

Storage Adapter

IBM libraries Not required

IBM storage and platform libraries

IBM storage and platform libraries

SAP LVM Edition

Standard Edition

(advanced scenarios

require Enterprise

Edition)

Standard Edition

Standard Edition

Standard Edition for NFS and GPFS Enterprise Edition for SAN based storage

Enterprise Edition

Virtu

aliz

ation M

an

age

me

nt IBM Power

Systems with IBM Systems Director

Yes, requires VMControl Express Edition

Yes, requires VMControl Express Edition

Yes, requires VMControl Enterprise Edition and

Storage Control

Not required Not required

IBM PureSystems with IBM Flex System Manager

IBM Power Systems with HMC

Yes Yes No

Sto

rage M

an

age

ment IBM Tivoli

Storage FlashCopy Manager

Not required Not required Not required Not required Required

Supported Storage

Any Any Any

NAS (NFS) GPFS IBM XIV IBM SVC IBM V7000 IBM DS8000

IBM XIV IBM SVC IBM V7000 IBM DS8000

Table 3 Overview of the Scenarios

2.2.4 Custom Cloning Capability and custom hooks/ operations

SAP LVM 2.0 introduced a new “custom cloning” feature. This feature allows creating a

system clone based on existing cloning procedures and integrating those into SAP LVM. One

can replace the pre-defined sub-steps in the “standard” workflow of the pre-clone, clone, and

post-clone steps by custom defined web services or scripts executed on the source and target

systems.

The web services/scripts need to be registered as a “Provider Implementation Definition” and

references the script/web services execution triggered by the SAP Host Agent on the source or



target system during the workflow step. The provider implementation definition specifies

“what” is executed on the host or on the SAP instance.

When the provider implementation definitions is done, the new custom provisioning process

then maps each provider implementations instead of the “standard” step of the built-in cloning

process. It specifies “when” and “under which circumstances” the provider implementation

definition is executed.

Provider implementation definitions can be used in the context of custom hooks or custom

operations too:

Custom hooks allow to add additional pre-, post-, or error processing steps to the

standard workflows defined in SAP LVM.

Custom operations allow triggering of user-defined activities directly from the SAP

LVM GUI or as part of mass operations. For example, the SAP Basis administrator

could run certain custom-defined scripts with privileged permissions on the hosts

without granting him “root” access in general.

2.3 Planning Project Resources and Timing

Table 4 summarizes the activities in order to implement a SAP LVM managed POWER

landscape comparable to our proof of concept setup described in the following section 2.4.

The estimated duration per phase provides a rough guideline only. In detail, the times are

dependent on the complexity of the SAP landscape (IT and SAP systems) and its operation as

well as on the availability of all the required skills for all involved disciplines.



Project Phase Included Tasks / Accomplishments Estimated time

Planning workshop, Verification of landscape prerequisites in client environment

Network and disk design requirements, SW prerequisites

Validate that customer has obtained the necessary SAP and AIX software, validate that dedicated IBM hardware infrastructure is available

SAP Landscape needs to support virtual host names, and a proper range of IP addresses needs to be available

2-3 Day workshop with customer

Plan and Prepare setup of storage infrastructure

Storage LUNs

Network, SAN zoning

Eventually map SVC to existing storage

Plan and create filesystems

3-4 days working with customer

Installation and configuration steps for the management environment (IBM Systems Director, SAP NetWeaver Landscape Virtualization Management)

Define LPARs on the managed systems, and install AIX OS on the LPARs

Set-up virtualization environment o Install and configure IBM Systems

Director including VMControl and Storage Control

o Integrate NIM server into System Director VMControl environment

o Set-up Storage Control for all storage entities (SAN, Storage System)

Install and configure storage management software (FlashCopy Manager, Storage CLI) on the management system

Plan and configure shared infrastructure services

o Install and configure LDAP (if not to be integrated with existing LDAP environment

o Plan for DNS

Install SAP LVM system

Two weeks working with customer

Installation and configuration steps of managed environment incl. basic testing Installation from scratch for one SAP System

Define AIX “golden image” content

Install first AIX LPAR with golden image

Install and configure storage management software (FlashCopy Manager, SVC CLI)

Install first “source” SAP system

LVM base configuration (Virtualization Manager, Storage Manager configuration, discover hosts, SAP systems, users)

Capture image in Systems Director VMControl for further deployment

Additional LVM use case specific configuration (storage, post processing, steps per use case) Note: configuration of PCA post processing requires skilled SAP specialist with both SAP Basis and Application/Business Process knowledge)

Two weeks with customer

Installation and configuration steps of managed environment incl. basic testing Migrate/ Customize one already existing SAP system

Adapt existing SAP system to virtualised environment

Eventually Upgrade SAP Host Agent

Install and configure agents and storage management

Install and configure storage management software (FlashCopy Manager, SVC CLI)

Discovery and setup of LPARs and SAP

At least 2 weeks with customer



Project Phase Included Tasks / Accomplishments Estimated time

systems in IBM Systems Director and SAP LVM

Additional LVM use case specific configuration (storage, post processing, steps per use case) Note: configuration of PCA post processing requires skilled SAP specialist with both SAP Basis and Application/Business Process knowledge)

Verification Steps using the customer’s SAP system

Perform functional verification

Execute and document each use case o Start / Stop / Relocate of LPARs and SAP

systems o Clone SAP Systems (target LPAR available) o Copy SAP Systems (target LPAR available) o Refresh SAP system o Clone SAP Systems (create target LPAR

from scratch) o Copy SAP Systems (create target LPAR

from scratch)

At least 2 weeks working with customer

Acceptance Tests

Support customer in LVM acceptance testing from basis or application level

At least 1 week

Document Solution or Handover

at least 2 days for handover, documentation at least 1 week

Table 4 Project Phases and estimated time

2.4 Tested configuration

The installation hints in this document are based on a proof of concept installation in the

ISICC demo landscape using IBM PureFlex Systems with Power nodes and IBM SAN

Volume Controller Storage as well as IBM XIV Storage Systems.

Figure 7 illustrates the environment in the proof of concept.

In the management environment:

Virtualization Manager is IBM Flex System Manager 1.3.0. including VMControl

2.4.3.1 and Storage Management 6.3.3.1.

SAP NetWeaver Landscape Virtualization Management software V2.0 SP02 is

deployed on top of a SAP NetWeaver Java AS 7.31 engine in a second partition.

Additional infrastructure servers like DNS, LDAP server, and NIM server are

available and configured.

In the managed environment

at least one SAP source system is existing

all hosts (AIX LPARs) for both the SAP source system(s) and SAP target system(s)

are registered within SAP LVM. The hosts are attached via virtual fibre channel to an

IBM XIV Storage system.

On each host the software components

SAP Host Agent package


IBM XIV XCLI



are installed and configured.

Figure 7 Architecture overview diagram

Table 5 reflects the detailed software stack tested in the scenario:

Software Version

SAP Landscape Virtualization Management

Software 2.0 SP02

IBM Flex System Manager / VMControl /

Storage Management 1.3.0 / 2.4.3.1 / 6.3.3.1

IBM Script Package

SAP Host Agent 7.20 Patchlevel 164

SAPINST (Rename)1

SAP Software Provisioning Manager

(SWPM) V1.0 SP4_2

AIX 7.1 TL2 SP2 7100-02-02-1316

IBM Tivoli Storage FlashCopy Manager 3.2.0.1

IBM XIV 10.2.4.a

XIVGUI XCLI 3.1 build 18

DB2 UDB DB2 v10.1.0.2 (Fixpak 2)

Table 5 Software Stack

2.5 SAP LVM Environment

2.5.1 Storage Virtualization

A SAP LVM infrastructure requires a centralized storage solution. The storage infrastructure

in the ISICC sample environment is based on IBM System Storage™ solutions using a

Storage Area Network (SAN).

1 Starting with SAP NetWeaver Landscape Virtualization Management 2.0 Support Package 03 the delivery of

software component "SAP SYSTEM RENAME 1.0" is part of the "SOFTWARE PROVISIONING MANAGER

1.0" package.


NIM Server AIX

Management Environment

IBM XIV SVC / Storwize

SAP LVM AIX

LDAP IBM Tivoli Directory Server

Source Target

VIOS

vFC (NPIV)

VM Control, Storage Manager

DNS

SAP NetWeaver Landscape Virtualization Management SAP NetWeaver Java AS

SAP Source AIX SAP Source

AIX SAP Source AIX SAP Source

AIX SAP Source AIX SAP Source

AIX SAP Component - SAP NetWeaver

Managed Environment

VIOS

vFC (NPIV)

SAP Target AIX

• Storage connected via vFC (NPIV)

• Compliant Storage Layout

IBM Tivoli FlashCopy Manager Storage Manager native Interface e.g. IBM XIV XCLI

IBM Tivoli FlashCopy Manager Storage Manager native Interface e.g. IBM XIV XCL

IBM Tivoli FlashCopy Manager Storage Manager native Interface e.g. IBM XIV XCLI



IBM System Storage virtualization products achieve the abstraction from physical volumes of

data storage to a logical level. It addresses the increasing complexity of managing storage,

while reducing the associated costs. Its main purpose is the full exploitation of the benefits

promised by a SAN. Virtualization enables data sharing, ensuring higher availability,

providing disaster tolerance, improving performance, allowing for consolidation of resources,

providing policy-based automation, and much more besides, which do not automatically result

from the implementation of today’s SAN hardware components. Storage virtualization is

possible on several levels of the storage network components, meaning that it is not limited to

the disk subsystem. Virtualization separates the representation of storage to the operating

system and its users, from the actual physical components.

Storage virtualization accumulates the storage into storage pools, which are independent of

the actual layout of the storage (that is, the overall file system structure). Because of this

independence, new disk systems can be added to a storage network, and data migrated to

them, without causing disruption to applications. Since the storage is no longer controlled by

individual servers, it can be used by any server as needed. In addition, it can allow capacity to

be added or removed on demand without affecting the application servers. Storage

virtualization will simplify storage management, which has been an escalating expense in the

traditional SAN environment.

IBM SAN Volume Controller (SVC), IBM Storwize V7000, IBM System Storage DS8000, or

IBM XIV System Storage is not mandatory for all the use-cases in an adaptive computing

environment, but they help in the overall management of the storage landscape. For the SAP

System Copy use-cases however the Storage Adapter and Storage Library require one of those

systems.

2.5.1.1 SAN Volume Controller System Storage

The IBM (SVC) is an in-band, block-based virtualization product that minimizes the

dependency on unique hardware and software, decoupling the storage functions expected in a

SAN environment from the storage subsystems and managing storage resources. SVC

combines software and hardware into a comprehensive, modular appliance that uses

symmetric virtualization.

Symmetric virtualization is achieved by creating a pool of managed disks (MDisks) from the

attached backend storage systems. Those storage systems are then mapped to a set of volumes

for use by attached host systems. System administrators can view and access a common pool

of storage on the storage area network (SAN). This functionality helps administrators to use

storage resources more efficiently and provides a common base for advanced functions.

Key feature exploited for the SAP System Copy scenarios is the ability to create and manage

FlashCopies of existing (SAP application) volumes. IBM FlashCopy is a part of the SVC

Copy Services and provides a point-in-time copy function. This includes thin-provisioned

FlashCopy to make multiple targets affordable.

2.5.1.2 IBM Storwize V7000 Storage System

The IBM Storwize V7000 is a storage server with internal disks, providing the same storage

virtualization functions and features like SAN Volume Controller.

2.5.1.3 IBM System Storage DS8000

The IBM System Storage DS8000 is a high-performance, high-capacity, secure storage

system designed to deliver the highest levels of performance, flexibility, scalability, resiliency



and total overall value for the most demanding, heterogeneous storage environments. SAP

LVM 2.0 supports this IBM storage solution since service pack 3.

2.5.1.4 IBM XIV System Storage

The IBM® XIV® Storage System is a high-end disk storage series designed to address

storage challenges across the application spectrum, including for virtualization, email,

database, and analytics and data protection solutions. The IBM XIV Storage System is a fully

virtualized system designed to eliminate the need for performance tuning and numerous other

storage management activities. IBM XIV provides consistent, hotspot-free enterprise

performance and exceptional ease of use. As virtualized storage that meshes tightly with

hypervisors, XIV offers optimal agility for cloud and virtualized environments.

Key feature exploited for the SAP System Copy scenarios is the ability to create and manage

snapshots of existing (SAP application) volumes. The IBM XIV Storage System has taken

this concept one step further, offering a completely innovative approach to snapshot creation

and management. The XIV system offers clear advantages, including the following:

Unlimited number of snapshots in the system.

Snapshot creation in virtually zero time, regardless of the size of replicated volumes.

Unaffected performance levels in a system that supports snapshots, regardless of the

number of snapshots currently defined in the system.

2.5.1.5 General Parallel File System (GPFS)

Exploiting GPFS is the easiest and most convenient way to implement the „classic“ adaptive

scenarios for relocating SAP instances between different hosts.

IBM's General Parallel File System (GPFS) provides file system services to parallel and serial

applications. GPFS allows parallel applications simultaneous access to the same files, or

different files, from any node, which has the GPFS file system mounted, while managing a

high level of control over all file system operations. GPFS is particularly appropriate in an

environment where the aggregate peak need for data bandwidth exceeds the capability of a

distributed file system server.

GPFS allows users shared file access within a single GPFS cluster and across multiple GPFS

clusters. A GPFS cluster consists of:

AIX nodes, a node may be:

o An individual operating system image on a single computer within a cluster

o A system partition containing an operating system

Network shared disks (NSDs) created and maintained by the NSD component of

GPFS

A shared network for GPFS communications allowing a single network view of the

configuration (a single network is used for GPFS communication, including the NSD

communication)

All disks utilized by GPFS must first be given a globally accessible NSD name. On AIX

server running GPFS, an existing virtual shared disk or physical disk may be given an NSD

name.



2.5.2 SAP NetWeaver Landscape Virtualization Management

All the base considerations are described in the SAP Notes Note 1783702 - SAP NetWeaver

Landscape Virtualization Management 2.0 and Note 1833980 - Integration of IBM

Components into SAP LVM 2.0.

SAP LVM distinguishes in its configuration between:

Instances

An instance is a part of an SAP System (database instance, central services instance,

application server instance). The SAP LVM use-cases typically are executed on instances

or whole SAP systems.

Hosts

A host describes an operating system installed on a physical or virtual server. SAP

instances may be relocated between hosts, or a host may be the target for a new SAP

system created via SAP system copy or clone. A new host may be deployed (via SAP

LVM and VMControl) so that it can be used as a target for a relocate operation or a SAP

system copy, clone, or refresh. SAP LVM monitors hosts through SAP Host Agents

running on the operating systems.

2.5.2.1 SAP LVM Virtualization Management

SAP LVM interacts with the virtualization managers to

extend host monitoring data available via SAP Host Agents with information about

virtualization layers (physical hosts and virtual machines)

provision new virtual machines

perform load balancing by starting, stopping and relocating virtual machines

SAP LVM includes two virtualization managers (adapters) for IBM solutions

IBM Flex Systems Manager/ VMControl Adapter

IBM Hardware Management Console for IBM Power

Both of them enable

Monitoring of IBM Power physical servers and logical partitions (LPARs)

Management of LPARs (Power on / Power off / Shutdown OS)

Relocation of LPARs using Live Partition Mobility

Additionally the “IBM Flex System Manager/VMControl” adapter provides provisioning

capabilities for both AIX OS and Storage.

See SAP 1833980 - Integration of IBM Components into SAP LVM 2.0 for detailed

description of supported configurations and features.

2.5.2.2 SAP LVM Storage Management

SAP LVM interacts with the storage managers (adapters) to

clone, copy, or refresh SAP systems

prepare, un-prepare, and relocate of SAP instances and systems

SAP LVM also includes an “IBM Storage Adapter” which supports invocation of

FlashCopies/SnapShots within the storage system to provide a fast, efficient, and application-

aware cloning of the SAP source system storage volumes.

https://service.sap.com/sap/support/notes/1783702







These functionalities are used in SAP system clone, copy, and refresh operations. In addition,

this adapter provides the required functionality to attach and detach storage volumes and

filesystems to and from the LPARs during SAP system prepare, unprepared, and relocate

activities.

“IBM Storage Adapter” in SAP LVM 2.0 supports following storage system types:


IBM Storwize V7000 storage system

IBM SAN Volume Controller (incl. any supported backend storage system)^

IBM System Storage DS8000

See SAP Note 1833980 - Integration of IBM Components into SAP LVM 2.0 for detailed

description of supported configurations and features.

In this proof of concept we used the following configuration for SAP System Copy use cases:

IBM SAN Volume Controller 7.1.0.4

IBM XIV Storage System, Version 10.2 with XIV CLI 3.0

2.5.2.2.1 SAN Zoning Requirements

As a prerequisite for all the SAP System Copy use cases (SAP System clone, copy, refresh)

SAN based storage is required for usage with the “IBM Storage Adapter”.

SAN zoning on the target LPAR must be pre-configured. This can be easily fulfilled if the

volume(s) for the AIX operating system (rootvg) of the target LPAR is located on the same

storage like the SAP systems in scope of the System Copy.

Then the existing access to the rootvg on the target LPAR provides SAN access for the SAP

application volumes too. In this case, the “host” and “adapter id” definitions in the storage

system exist already, and the LPAR has an already zoned access in the SAN to the storage

system. The SAP application volumes are assigned to the LPAR by “LUN masking”

commands from the SAP LVM storage adapter.

The storage system should be zoned according to “single initiator zones”. Each zone set

should contain one (active) virtual fibre channel host port only, but can include multiple

storage adapter ports. (e.g. between three and six adapter ports for a storage system).

Although not mandatory it is best practise to define the name of the host in the storage system

identical to the name of the LPAR or its hostname. If virtual fibre channel adapters are used,

then all the WWPNs need to be included in port definition on the storage system, as well as in

the SAN zoning definition. This includes all “active” WWPNs via the VIOS pair, as well as

the secondary WWPNs used during a Live Partition Mobility relocation event.

2.5.3 Considerations for the SAP LVM hosts

Several storage attachment considerations need to be obeyed, and additional software

components need to be installed on the LPARs used as SAP LVM hosts. All these are

described briefly in this section.




2.5.3.1 VIOS / PowerVM and SAN attachment

For all the SAP System clone, copy, and refresh scenarios that invoke storage-based cloning,

all the storage volumes for both the source and the target systems need to be attached either

via virtual Fibre Channel (vFC) adapters with NPIV or via physical fibre channel adapters

attached directly to these systems.

In the proof of concept described in this paper we used NPIV attachment via two redundant

VIO servers. Each client LPAR is defined with two virtual fibre channel adapters. On each

VIO server a virtual fibre channel server adapters is created and assigned to the corresponding

client LPAR. Two pairs of WWPNs are created (WWPN1, WWPN2, plus the additional

WWPNs for the LPM scenario). We use single initiator zones. Each zone contains one

(active) virtual fibre channel host port and between three and six storage system ports. The

maximum number of paths that can be zoned to an AIX LPAR is 32. However there is no

benefit in having more than 24 paths.

Figure 8 illustrates the redundant attachment of LPARs with IBM XIV storage system via two

redundant VIO Servers. Each of the VIO servers has one or more physical dual-port fibre

channel adapters assigned to the partition. The two ports of the fibre channel adapter card are

connected to two different SAN fabrics. Also, the storage system is connected to both fabrics.

If an LPAR has only one fibre channel adapter, then zone it to three IBM XIV modules. Zone

it to six modules only in case the LPAR has very high throughput requirements.

Figure 8 Redundant virtual fibre channel attachment

2.5.3.2 AIX operating system specifics

The AIX operating system on the managed systems needs to fulfil certain criteria:

IPsec

IPsec is required for network fencing on target hosts for SAP system clone, copy or

refresh scenarios and needs to be activated in the OS.

Domain Name System

For the sake of simplicity, all the virtual hostnames of the SAP instances and systems

LPAR

IBM XIV Storage

connected via vFC

(NPIV)

vFC Client

WWPN1

vFC Client

WWPN2

vFC Client

WWPN3

vFC Client

WWPN4

VIOS #1 VIOS #2

vFC ServervFC Server vFC ServervFC Server

FC adapter

port 1

FC adapter

port 0

FC adapter

port 1

FC adapter

port 0

FC adapter

port 1

FC adapter

port 0

FC adapter

port 1

FC adapter

port 0

Fabric #1

Fabric #2



should be provided by DNS. One may also configure DNS update functionality in

SAP LVM. Local hostname resolution using “/etc/hosts” must be configured as

alternative path.

Central User Management is recommended

In the test environment, LDAP is used for that purpose. The LDAP filesets need to be

installed, and the OS needs to be configured as LDAP client.

Services entries

We used LDAP as a central repository for the SAP related service entries too.

Otherwise an /etc/services file need to be prepared and distributed to all hosts.

2.5.3.3 IBM Tivoli Storage FlashCopy Manager

IBM Tivoli Storage FlashCopy Manager software provides fast application-aware backups

and restores, so leveraging advanced snapshot technologies in IBM storage systems. The

software provides following features in general:

Near-instant application-aware snapshot backups, with minimal performance impact

for IBM DB2, Oracle, SAP, Microsoft SQL Server, and Exchange

Improvement in application availability and service levels through high-performance,

near-instant restore capabilities that reduce downtime

Integration with IBM Storwize V7000, IBM System Storage DS8000, IBM System

Storage SAN Volume Controller and IBM XIV Storage System on AIX, Solaris,

Linux, and Microsoft Windows

Fulfilment of advanced data protection and data reduction needs with optional

integration with IBM Tivoli Storage Manager

In the context of SAP landscape management, IBM Tivoli FlashCopy Manager is utilized to

create instantaneously a consistent copy of an online SAP system.

2.5.3.4 Storage Management Interfaces

2.5.3.4.1 XCLI

The IBM XIV Storage System command-line interface (XCLI) provides a mechanism for

issuing commands to manage and maintain the XIV storage systems. XCLI commands are

entered on the XCLI client either by a user or by another application. The XCLI client and the

XIV system communicate using Secure Sockets Layer (SSL) over TCP/IP. . Authentication is

based on username/password.

2.5.3.4.1 SAN Volume Controller Command Line Interface (SVC CLI)

The SAN Volume Controller command-line interface (SVC CLI) provides a mechanism for

issuing commands to manage and maintain the IBM SVC storage system. Communication is

achieved by SSH using SSL over TCP/IP: The SAN Volume Controller cluster acts as the

SSH server in this relationship. The SSH client provides a secure environment to connect to

the remote machine based on public and private keys for authentication.

2.5.3.4.2 SMI-S

The IBM SVC’s CIMOM interface is supplied with the SVC Master Console and is

automatically installed as part of the SVC Master Console installation. For the IBM DS8000,

the CIMOM interfaces is supplied with the HMC. The CIMOM interface translates an SMI-S



(Storage Management Initiative Specification) command into a proprietary command that the

device understands and then convert the proprietary response back into the SMI-S based

response.

2.5.3.5 SAP Host Agent

The SAP Host Agent package contains all the required elements for centrally monitoring of

any hosts. It is used by SAP LVM to monitor and manage SAP instances and hosts. The SAP

Host Agent package needs to be installed and configured on each host containing an SAP

component managed by the SAP NetWeaver Landscape Virtualization Management software.

The SAP Host Agent package is automatically installed as part of the “SAPinst” installation

procedure for any components of SAP NetWeaver 7.0 EHP2 and higher. However it may

need to be updated to a higher patch level to exploit all SAP LVM management features. It is

also required to install SAP Host Agent manually on operating systems without any SAP

systems.

In our case these are operating systems that act as targets in SAP system clone, copy, refresh,

or relocate operations.

2.5.4 Considerations for the SAP LVM Instances and Systems

All SAP systems need to be installed compliant to a SAP LVM environment. This includes

different considerations for user management, network, and storage. Some best practises are

discussed in the following.

2.5.4.1 User Management

Ensure that the administrative users are available with same UID and GID on all the LPARs

involved in the scenario. A central user management solution provides this requirement and

we used LDAP for this purpose.

2.5.4.2 Network Considerations

The following guidelines for the network setup for the SAP system to be managed by SAP

LVM needs to be considered:

Provide a dedicated virtual hostname for each SAP instance

The various instances of an SAP system (SAP database, SAP Central Services, SAP

instances) shouldn’t share an IP address. Each instance should own a dedicated IP

address.

Provide a dedicated IP address for each virtual host name

Do not use multiple virtual hostnames (IP alias names) referring to one and the same

IP address.

Ensure that all the virtual host names can be resolved on the SAP LVM system and on

all hosts

Ensure that the reverse lookup on the SAP LVM server and all other hosts provides a

one-to-one mapping

2.5.4.3 Storage Layout

Separate all storage volumes for the different instances

During the prepare, unprepared, and relocate activities is must be possible to handle

the storage independent of the instances:

o Create different AIX volume group for the filesystems related to the SAP

Central Instance and the Database



o If a Solution Manager Diagnostics agent exists for the SAP System, place its

filesystems to the Central Instance.

o If there is a dedicated Solution Manager diagnostics agent for the Database,

then its filesystems need to be part of the volume group of the database

o If you intend to use FlashCopy backup for the production database also, then

split the database content into three VGs: database data (tablespaces), database

online logs, and other database files (instance directory, executables, archive

logs etc …)



3 Installation

The following table provides an overview on the operational model. In the test environment,

both the management environment and the managed environment are hosted on IBM AIX

partitions. Most of the deployment units for the managed environments were directly added to

the AIX operating system image. They were part of the NIM deployment process for a new

SAP LVM host and automatically available on new systems.


Node Deployment Unit

Management LPAR SAP LVM

AIX

LDAP Client (IBM Tivoli Directory Server Client)

SAP NetWeaver 7.3 Java System / DB2 UDB

SAP NetWeaver Landscape Virtualization Management 2.0 SP02

IBM Tivoli Storage FlashCopy Manager [acsd] (including GSKit 8 package)

XIVGUI XCLI

SAP Host Agent package and SAP Adaptive Computing Extensions (SAPACEXT)

Management LPAR IBM Flex System Manager

AIX

LDAP Client (IBM Tivoli Directory Server Client) optional

IBM Flex System Managers incl. VMControl plug-in

IBM Flex System Manager Storage Management (including embedded Storage Control plug-in)

NIM Server

AIX, setup as NIM Master

LDAP Client (IBM Tivoli Directory Server Client) optional

IBM Flex System Manager VMControl subagent for NIM

LDAP Server AIX

IBM Tivoli Directory Server

DNS Server AIX bind

Managed Environment

Node Deployment Unit Part of “AIX

image template”

on all SAP LVM hosts

AIX Yes

LDAP Client (IBM Tivoli Directory Server Client) Yes

SAP Host Agent package and SAP Adaptive Computing Extensions

Yes

IBM Tivoli Storage FlashCopy Manager [acsgen] (including GSKit 8 package)

Yes

XIVGUI XCLI (for IBM XIV Storage Systems)

Yes

SAP System Rename (required for SAP System Copy on target host)

No (accessed

from a central NFS share)

SAP Software Provisioning Manager (required for automated provisioning of SAP Application Server)

on one SAP LVM host

SAP ECC 6.0 source system DB2 UDB

SAP PostCopyAutomation tool (required for SAP System Copy and Refresh)

Table 6 Operational Model

This chapter describes the installation of the components for both the management and

managed environments for an AIX/ POWER environment. Not all the installation steps will

be described in all details; it is assumed that the reader has sufficient background knowledge

to perform these steps.



In principle the SAP NetWeaver Landscape Virtualization Management server could be

deployed on another platform, e.g. Linux/Intel or Microsoft Windows/Intel. Then the

deployment unit “IBM Tivoli Storage FlashCopy Manager [acsd]” would need to be moved to

another AIX server (e.g. to the NIM server). XIVGUI XCLI, SAP Host Agent package, and

SAPACEXT libraries then would need to be installed on this server too.

3.1 Naming Conventions

In a SAP LVM environment SAP systems can potentially run on any SAP LVM enabled

server. For example, one could install all SAP system relevant data (database, binaries, and

working directories) on a distributed file system. In this case, we recommended putting this

data under a common root directory on that distributed file system.

A careful planning is required to avoid the collision of various names like IP names, user

names, user ids and group ids, directory structures, etc.

One method is to include the SAP System ID (SID), which typically is unique in a customer

environment, in the required SAP system IP names and root directories of the SAP system

data directories.

In addition, if you want to run multiple SAP systems or instances on a single compute node,

you have to plan for unique SAP instance numbers as well, as you cannot run two SAP

instances with the same instance number (but different SIDs) within the same OS image.

3.1.1 Sample Instance IP Names for SID C01

The following is one possible naming convention to build SAP systems IP names. There are

no specific requirements for IP names in the SAP LVM infrastructure, however choosing and

sticking to one convention will help to simplify the administration.

c01dbs IP name for database server

c01scs IP name for SAP central services instance

c01pas IP name for SAP primary application server instance

c01as1 IP name for additional application server instance

3.1.2 Sample directory structure for SID C01 on a Distributed File System

Each SAP system should have a common high level subdirectory name to provide a standard

entry point to all data that belongs to the same system. For example if the distributed file

system is mounted at /sapfs, then all data belonging to SAP system C01 could be allocated

under the subdirectory /sapfs/sapC01. The “root” directory for the SAP system C01 would

then be /sapfs/sapC01. In that root directory, all instance relevant subdirectories are stored.

The mappings between the physical locations on the storage subsystem and the directory

locations as they are expected by the SAP kernel have to be defined in SAP LVM. The

controller will then create the required mapping between the new data location and the

required standard path before starting an instance on a server. Table 7 shows a sample

directory structure for SID C01:

Directory Content Path in standard

installation

<root>/home/c01adm Home directory for SAP admin user /home/c01adm

<root>/db2 <root>/oracle

Database server, client, instance /db2 /oracle

<root>/sapmnt/C01 Directory for executables, profiles, shared files /sapmnt/C01

<root>/usr/sap/c01scs Instance directories for central services instance /usr/sap/C01

<root>/usr/sap/c01as1 Instance directories for application server 1 /usr/sap/C01

/sapfs/saptrans01 Common transport directory /usr/sap/trans



Table 7 Sample directory structure for a distributed file system implementation

The screenshot in Figure 9 shows the SAP instance configuration screen in SAP LVM, which

illustrates the mapping between physical layout and the required layout for the SAP database

instance EC1.

The storage type is Distributed File System (DFS) with a subtype of “gpfs", which refers to a

General Parallel File System installation.

The entries with mount option dirmount define the required mapping between the physical

location and the mount point that is expected by the SAP kernel.

Figure 9 System configuration in SAP LVM with distributed file system layout

3.1.3 Sample directory structure using Network Attached Storage (NAS)

The same concept as in the previous section can be used to layout the physical data locations

on a NAS device. A typical organization unit with NAS is called a volume where you store

directories and files. Those resources then have to be exported to the clients that need to

access them. The root directory for SAP system C01 could for example look like

nas_server:/vol/sapvol1/sapC01, where nas_server is the IP name of the NAS device. Again

the mappings between physical locations and expected locations have to be defined in the

SAP LVM Controller. The controller will then create the required mappings (in this case

using client NFS mount instead of creating symbolic links) before starting a instance on a

server.

3.2 Install Storage Infrastructure

3.2.1 Exploit GPFS as “adaptive storage”

This section provides a brief summary of setup tasks to get started with a GPFS cluster using

the ISICC SAP LVM landscape implementation as an example.

Using GPFS it is possible to carry out the use cases of the standard edition of LVM including

start, stop, and relocate of SAP systems. The use-cases for System Clone, Copy, and Refresh

are currently not supported with GPFS.

We do not cover in detail typical GPFS administration tasks like GPFS installation and

administration, the creation and management of storage pools, definition of file placement

policies, etc. For those tasks you need to refer to the standard product documentation and

available Redbooks. Especially to configure data availability and disaster recovery scenario,

you should carefully review the GPFS Base documentation (e.g. the “GPFS Concepts,

http://publib.boulder.ibm.com/infocenter/clresctr/vxrx/index.jsp

http://publib.boulder.ibm.com/epubs/pdf/a7604135.pdf



Planning, and Installation Guide” and the “GPFS Advanced Administration Guide”).

Additional information can be found in the GPFS Wiki

3.2.1.1 Planning for GPFS

3.2.1.1.1 Quorum Nodes

The first task is to decide on a node quorum strategy. As described in the “GPFS Concepts,

Planning, and Installation Guide”, GPFS enforces a node quorum rule to prevent multiple

nodes from assuming the role of the file system manager in the event of a network

communication problem in the cluster. A majority of quorum nodes must remain active in

order for the cluster to sustain normal file system usage (multi-node quorum).

The ISICC implementation uses a new node quorum mechanism that was introduced with

GPFS V2.3 – node quorum with tiebreaker disks, which allows you to run with as little as one

quorum node available as long as you have access to a majority of the quorum disks. Please

refer to Chapter 2 in the “GPFS Concepts, Planning, and Installation Guide” for a detailed

description of the quorum concept and guidelines to select quorum nodes. In the ISICC setup,

we use two quorum nodes and one tiebreaker disk.

3.2.1.2 Install and Setup GPFS

3.2.1.2.1 Install GPFS Software

You need to install the following filesets:

gpfs.base

gpfs.msg.en_US

This is best accomplished by including these filesets in a NIM installp bundle.

3.2.1.2.2 Create GPFS Cluster

The mmcrcluster is used to create a GPFS cluster. Our setup uses two quorum nodes: siccps10

and siccps12. The same nodes are also used as primary (option –p) and secondary (option –s)

configuration server to store the GPFS cluster configuration data. For cluster communication

we use ssh and scp instead of rsh and rcp, which has to be specified with the –r and –R

options. The –A option specifies that GPFS daemons are to be automatically started when

nodes come up. The default is not to start daemons automatically. Our cluster is named

siccfs.isicc (-C option).

root> mmcrcluster –N siccps12:manager-quorum,siccps10:manager-quorum –p siccps12 –s siccps10 –

r /usr/bin/ssh –R /usr/bin/scp –C siccfs.isicc –A

After a successful creation of the cluster, you can verify your setup using the mmlscluster

command.

root> mmlscluster

GPFS cluster information

========================

GPFS cluster name: siccfs.isicc

GPFS cluster id: 691765743288319456

GPFS UID domain: siccfs.isicc

Remote shell command: /usr/bin/ssh

Remote file copy command: /usr/bin/scp

GPFS cluster configuration servers:

-----------------------------------

Primary server: siccps12

Secondary server: siccps10


http://publib.boulder.ibm.com/epubs/pdf/c2351825.pdf

http://www.ibm.com/developerworks/wikis/display/hpccentral/General+Parallel+File+System+%28GPFS%29






Node Daemon node name IP address Admin node name Designation

----------------------------------------------------------------------------------------------

1 siccps12 9.153.165.71 siccps12 quorum-manager

2 siccps10 9.153.165.69 siccps10 quorum-manager

Then you need to startup GPFS on all nodes and you should check the state of the cluster:

root> mmstartup -a

root> mmgetstate -aL

Node number Node name Quorum Nodes up Total nodes GPFS state Remarks

------------------------------------------------------------------------------------

1 siccps12 2 2 8 active quorum node

2 siccps10 2 2 8 active quorum node

3.2.1.2.3 Define Network Shared Disk (NSD) Devices

GPFS uses Network Shared Disk (NSD) devices to provide access to the disks. A NSD device

can either be physically attached (using a SAN infrastructure) to a node or it can be accessed

using a virtual connection (through the network) using a primary and, if specified, backup

NSD server. GPFS determines if a node has physical or virtual connectivity to an underlying

NSD through a sequence of commands invoked from the GPFS daemon. This determination

is called disk discovery and occurs at both initial GPFS startup as well as whenever a file

system is mounted.

In the ISICC proof of concept landscape, we used the following configuration file to create

the NSD devices:

/var/mmfs/conf/nsd.disks:

#Description of disk attributes

#<disk name>:<primary NSD server>:<2ndary NSD server>:<disk usage>:<failure group>:<NSD name>

#Tiebreaker disk

hdisk3:::descOnly:-1:nsd_tbrk_01:

#Data and metadata disk for /siccfs

hdisk4:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_01:

hdisk5:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_02:

The first disk (hdisk3) holds file descriptor data only and is used as a tiebreaker disk. hdisk4

and hdisk5 are designated as data disks for our LVM cluster file system. We use our quorum

nodes as primary and backup NSD servers as well, although all of our nodes do have physical

access to the disks. This provides some automatic fallback capability should the SAN access

become unavailable.

Create the NSD devices using the mmcrnsd command and verify your setup using the

mmlsnsd command (the option -v yes specifies that the NSD are to be created only if the disk

has not been formatted by a previous invocation of the mmcrnsd command):

root> mmcrnsd -F /var/mmfs/config/nsd.disks -v yes

root> mmlsnsd -aL

File system Disk name NSD volume ID Primary node Backup node

---------------------------------------------------------------------------------------------

siccfs nsd_siccfs_01 0999A547447D7EFE siccps12 siccps10

siccfs nsd_siccfs_02 0999A547447D7F00 siccps12 siccps10

(free disk) nsd_tbrk_01 0999A547447D7DA6 (directly attached)

The specified disk file is rewritten during NSD creation. Each hdisk entry is preceded by a

comment (# sign) and followed by the equivalent NSD entry. The new file can then be reused

for file system creation during subsequent configuration steps. The file system create

command uses NSD devices and not AIX hdisks.



3.2.1.2.4 Activate Node Quorum using Tiebreaker Disks

You need to shutdown the GPFS cluster and use the mmchconfig to activate the tiebreaker

disks. Use the following commands to activate the NSD device nsd_tbrk_01 as a tiebreaker

disk:

root> mmshutdown –a

root> mmchconfig tiebreakerDisks="nsd_tbrk_01"

root> mmstartup –a

You can use the mmlsconfig and mmgetstate commands to check your configuration after the

switch to tiebreaker mode:

root> mmlsconfig

Configuration data for cluster siccfs.isicc:

-------------------------------------------

clusterName siccfs.isicc

clusterId 691765743288319456

clusterType lc

autoload no

useDiskLease yes

maxFeatureLevelAllowed 903

tiebreakerDisks nsd_tbrk_01

[siccps12]

takeOverSdrServ yes

File systems in cluster siccfs.isicc:

------------------------------------

/dev/siccfs



------------------------------------------------------------------------------------

1 siccps12 1* 2 8 active quorum node


In the mmlsconfig output you can see that device nsd_tbrk_01 is used as tiebreaker disk. The

quorum information in the mmgetstate output is displayed as “1*”, which indicates that this is

a two node tiebreaker disk cluster.

3.2.1.2.5 Create GPFS File System

The last step is to create the file system itself. We used the following excerpt of the disk file

that was created by the mmcrnsd command as input to create file system command:

/var/mmfs/config/disks.siccfs:

#Description of disk attributes

#<disk name>:<primary NSD server>:<2ndary NSD server>:<disk usage>:<failure group>:<NSD name>

#Data and metadata disk for /siccfs

# hdisk4:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_01:

nsd_siccfs_01:::dataAndMetadata:1::

# hdisk5:siccps12:siccps10:dataAndMetadata:1:nsd_siccfs_02:

nsd_siccfs_02:::dataAndMetadata:1::

The mmcrfs command was used to create the /siccfs cluster file system:

root> mmcrfs /siccfs /dev/siccfs -F /var/mmfs/config/disks.siccfs –A automount -B 256K -v yes

We used the –A automount option to specify that the file system is automatically mounted

when its first accessed.

3.2.1.2.6 Mount GPFS File System

If you did not use the automount option during file system creation, you can use the standard

AIX mount command to mount the cluster filesystem:



root> mount /siccfs

3.2.1.3 Considerations for SAP Instance Installation on GPFS

A SAP system has to be up and running before it can be registered and configured in the

Landscape Virtualization Management software. SAP LVM is not able to create the required

environment (mount target directories) for a new installation, this has to be done manually

(see also section 3.4.5). You should first create the directory structure on the GPFS file

system according to your naming conventions and then mount the subdirectories at the mount

points that are expected by the SAP installation routines.

The following commands create a sample directory structure for system C01:

mkdir /siccfs/sapC01

mkdir /siccfs/sapC01/oracle

mkdir /siccfs/sapC01/sapmnt

mkdir /siccfs/sapC01/usrsap

mkdir /siccfs/sapC01/trans

mkdir /siccfs/sapC01/c01adm

Then you need to create the required mapping before you can start the installation. This is

done with a normal mount command using the option –v mmfs (you may have to create some

of the directories for the mount points first, e.g. mkdir /home/c01adm).

mount -v mmfs /siccfs/sapC01/oracle /oracle

mount -v mmfs /siccfs/sapC01/sapmnt /sapmnt/C01

mount -v mmfs /siccfs/sapC01/usrsap /usr/sap/C01

mount -v mmfs /siccfs/sapC01/trans /usr/sap/trans

mount -v mmfs /siccfs/sapC01/c01adm /home/c01adm

After the installation is finished and the new SAP instance is registered at the Solution

Manager, you should unmount all directories and proceed to the SAP LVM configuration

tasks.

3.2.1.4 Instance Configuration in SAP LVM for GPFS

When you enable SAP systems in SAP LVM, you have to configure the data storage on the

System tab in the Configuration menu (for further details see chapter 4.7).

Data residing on a GPFS file system must be defined with Storage Type DFS. The Partner Id

is ibm and FS/SRID Type is gpfs. Please note that these values are case sensitive and have to

be entered using lower case.

The Export Path designates the directories as they are stored in the GPFS file system and the

Mount Point/SRID is the mount point as expected by the SAP kernel. The last field is Mount

Options, the available options for GPFS file systems are fsmount and dirmount.

The first one can be used to mount the GPFS file system itself (in case you don’t use the

GPFS automount option) and the second option to specify the subsequent subdirectory

mounts.

For example to mount subdirectory /usr/sap/EC1 on a GPFS file system that does not use the

automount option, you would need the following two entries:

Storage Type DFS

Mount Point/SRID /siccfs

Export Path /dev/siccfs

Mount Options fsmount

FS/SRID Type gpfs



Partner ID ibm

Storage Type DFS

Mount Point/SRID /usr/sap/EC1

Export Path /siccfs/sapEC1/usrsap

Mount Options dirmount

FS/SRID Type gpfs

Partner ID ibm

Figure 10 shows an example configuration of a GPFS based SAP system within the ISICC

SAP LVM landscape. The database used in this example is Oracle. For DB2 based systems,

the filesystems need to be created and mounted accordingly to the DB2 file system structure.

Note that a separate pool for SAP systems using the GPFS file systems was created in SAP

LVM.

Figure 10 Example configuration of an SAP system using GPFS

3.2.1.5 Add a Node to GPFS Cluster

You need to add an AIX node to the GPFS cluster before you can mount the cluster file

system. This is done with the mmaddnode command after the GPFS software installation is

completed:

root> mmaddnode –N siccps18

You can remove a node from the cluster using the mmdelnode command and we recommend

that you remove the node before you start a NIM re-installation of a specific node.

You can list the state of cluster nodes with the mmgetstate command:



------------------------------------------------------------------------------------



3 siccps17 1* 2 8 active








3.2.2 SAN Storage

The SAP NetWeaver Landscape Virtualization Management allows the direct management of

natively attached storage without having the need of a shared file system like GPFS.

IBM provides a deep integration of IBM storage functionality with SAP LVM.

This includes, but is not limited to,

creation and mapping of storage volumes to hosts or LPARs

discovering the volumes within the operating system

create file systems

creation of snapshots of volumes

mapping the snapshots to hosts / LPARs

re-mapping of volumes

This allows automated and fast moving, cloning and copying of SAP systems.

This chapter briefly describes the storage components and setup used in this proof of concept.

You can find the detailed description about how the components were installed and

configured in chapter 3.4.4.

3.2.2.1 Storage Components

Storage systems:

IBM SAN Volume Controller


Firmware Version: 10.2.4a

79 TB usable space

A separate pool ‘ISICC_cloud’ was created for the volumes of the test environment

Storage Management Software:

XIV CLI version (XCLI) version: 3.1 build 18

IBM Tivoli FlashCopy Manager: 3.2.0.1

3.2.2.2 SAP LVM SAN Configuration and File Systems

You need to install the SAP instances to be managed with SAP LVM according to the SAP

installation guides.

The following file system layout is recommended, and required if IBM Tivoli® Storage

FlashCopy® Manager is used to create a FlashCopy backup or clone of a SAP DB instance;

typically this applies to SAP production systems. The number of sapdata file systems (see

chapter 3.4.5.1) might vary, the default is four. A minimum of three different volume groups

need to be configured, as shown in Figure 11.



Figure 11 Volume Group and Filesystem layout

List of file systems used: root> mount

node mounted mounted over vfs date options

-------- --------------- --------------- ------ ------------ ---------------

/dev/hd4 / jfs2 Jul 02 09:47 rw,log=/dev/hd8

/dev/hd2 /usr jfs2 Jul 02 09:47 rw,log=/dev/hd8

/dev/hd9var /var jfs2 Jul 02 09:47 rw,log=/dev/hd8

/dev/hd3 /tmp jfs2 Jul 02 09:47 rw,log=/dev/hd8

/dev/hd1 /home jfs2 Jul 02 09:48 rw,log=/dev/hd8

/dev/hd11admin /admin jfs2 Jul 02 09:48 rw,log=/dev/hd8

/proc /proc procfs Jul 02 09:48 rw

/dev/hd10opt /opt jfs2 Jul 02 09:48 rw,log=/dev/hd8

/dev/livedump /var/adm/ras/livedump jfs2 Jul 02 09:48 rw,log=/dev/hd8

siccserv /siccfs /home/siccfs nfs4 Jul 02 09:48 ro,bg,hard,intr,vers=4,sec=sys

/dev/lv10 /sapmnt/CL6 jfs2 Jul 02 11:02 rw,log=/dev/loglv00

/dev/lv11 /usr/sap/CL6 jfs2 Jul 02 11:02 rw,log=/dev/loglv00

/dev/lv13 /home/cl6adm jfs2 Jul 02 11:02 rw,log=/dev/loglv00

/dev/lv12 /sapinst jfs2 Jul 02 11:02 rw,log=/dev/loglv00

/dev/lv15 /usr/sap/DAA/SMDA97 jfs2 Jul 02 11:02 rw,log=/dev/loglv00

/dev/lv09 /db2 jfs2 Jul 02 11:02 rw,log=/dev/loglv01

/dev/lv16 /db2/CL6 jfs2 Jul 02 11:02 rw,log=/dev/loglv01

/dev/lv01 /db2/CL6/db2cl6 jfs2 Jul 02 11:02 rw,log=/dev/loglv02

/dev/lv03 /db2/CL6/log_dir jfs2 Jul 02 11:02 rw,log=/dev/loglv03

/dev/lv04 /db2/CL6/log_archive jfs2 Jul 02 11:02 rw,log=/dev/loglv01

/dev/lv02 /db2/CL6/db2dump jfs2 Jul 02 11:02 rw,log=/dev/loglv01

/dev/lv05 /db2/CL6/sapdata1 jfs2 Jul 02 11:02 rw,log=/dev/loglv02




/dev/lv14 /var/db2 jfs2 Jul 02 11:02 rw,log=/dev/loglv01



List of volume groups used:

root> lsvg -l vg00 vg01 vg02 vg03

vg00:

LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT

loglv00 jfs2log 1 1 1 open/syncd N/A

lv12 jfs2 256 256 1 open/syncd /sapinst

lv10 jfs2 251 251 1 open/syncd /sapmnt/CL6

lv11 jfs2 376 376 1 open/syncd /usr/sap/CL6

lv13 jfs2 8 8 1 open/syncd /home/cl6adm

lv15 jfs2 62 62 1 open/syncd /usr/sap/DAA/SMDA97

vg01:



lv14 jfs2 1 1 1 open/syncd /var/db2

lv02 jfs2 7 7 1 open/syncd /db2/CL6/db2dump

lv04 jfs2 254 254 1 open/syncd /db2/CL6/log_archive

lv09 jfs2 178 178 1 open/syncd /db2

lv16 jfs2 1 1 1 open/syncd /db2/CL6

vg02:



lv01 jfs2 1 1 1 open/syncd /db2/CL6/db2cl6

lv05 jfs2 313 313 1 open/syncd /db2/CL6/sapdata1




vg03:



lv03 jfs2 497 497 1 open/syncd /db2/CL6/log_dir

List of hdisks used:

root> lsvg -p vg00 vg01 vg02 vg03

vg00:

PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION

hdisk2 active 1019 65 00..00..00..00..65

vg01:


hdisk1 active 1019 577 77..00..92..204..204

vg02:


hdisk3 active 1279 25 00..00..00..00..25

vg03:


hdisk4 active 1019 521 113..00..00..204..204

The list of volumes defined on the XIV storage system for SID CL6 is shown in Figure 12.

It also shows the snapshots created during the cloning of SAP system CL6 as source system.



Figure 12 Volumes of SAP system CL6

The volume wwn is part of the hdisk unique_id.

This attribute is used to identify the hdisk on the storage subsystem, as highlighted in yellow

and green:

XIV 1300360>>vol_list vol=CL6_datavg_1 -t wwn

WWN

00173800016800CA

root> lsattr -l hdisk3 -E -a unique_id

unique_id 2611200173800016800CA072810XIV03IBMfcp

Figure 13 shows the database instance of SAP system CL6 in LVM and the according mount

points.



Figure 13 SAP System CL6 in LVM and mount points

We recommend that host names defined on the storage system to which the volumes are

mapped exactly match the real host/node name. The primary mapping of operating system to

the host name on the storage system is done by using port IDs (WWPNs). In case this is not

possible for some reason, mapping is identified by comparing the real host name of the

operating system with the host name on the storage system.

IBM Tivoli FlashCopy Manager (FCM) is used to manage the snapshots on the storage

systems for SAP system cloning and copying purposes and must be also installed on all SAP

LVM managed systems.

The installation and configuration steps of FCM are described in detail in chapter 3.4.4).

3.3 Install Management Environment

3.3.1 Install SAP NetWeaver Landscape Virtualization Management Software

The current SAP NetWeaver Landscape Virtualization Manager 2.0 software is a Java

application that is deployed on a SAP NetWeaver Application Server Java system of usage

type “Basic”.

Please download the appropriate installation guide for the target operating system and

database platform combination and follow the instructions in that guide to install a SAP

NetWeaver Java System:

http://service.sap.com/instguidesNW73


As always please pay particular attention to the SAP notes mentioned in the “SAP Notes for

the Installation” section of the installation guide.

Download and deploy the SAP NetWeaver Landscape Virtualization Management software

after the installation and configuration of the SAP Java System is complete.

Please download the latest version of the installation guide for your licensed version (standard

edition or enterprise edition):

http://service.sap.com/instlvm

Again it is important to check all SAP notes mentioned on the download page and in the

installation guide itself. The installation guide describes the necessary steps to install LVM

after the initial setup of SAP NetWeaver AS Java.

At a high level, the installation of a SAP LVM management system requires:






1. Install SAP NetWeaver AS Java with usage type “BASIC”

2. Download the required SCA files for SAP LVM from the SAP Service Marketplace

http://service.sap.com/swdc

Software DownloadsBrowse our Download CatalogSAP NetWeaver and

complementary productsSAP NW LANDSC VIRT MGT ENT

3. Deploy the downloaded SCA files using one of the supported implementation tools

(SWPM or telnet)

4. Perform the initial setup using the configuration wizard as described in the post-

installation section

The last step is described in detail on the SAP Help Portal at:

http://help.sap.com/lvmstd20

http://help.sap.com/lvment20

Initial SetupCarrying Out the Initial Setup Using the Configuration Wizard

Follow the procedure documented in the help portal to step through the configuration wizard.

After the wizard finished successfully, you should be able to access SAP LVM using the

following URL: http://<host>:<port>/lvm.

3.3.2 Install IBM Tivoli Storage FlashCopy Manager

SAP LVM exploits IBM Tivoli Storage FlashCopy Manager to perform snapshot backups for

file systems related to the SAP Systems. The FCM setup is done according to the FlashCopy

backup of "Custom application environments" documentation. The snapshots are managed as

backup versions using the IBM Tivoli Storage FlashCopy Manager management policies. The

storage volumes for the target system are acquired and mounted on the target LPAR.

The management agent (acsd) of Tivoli Storage FlashCopy Manager is installed centrally on

one of the management LPARs. The "acsd" agent coordinates the snapshot backup operation

and controls the backup flow and mediates between the other agents. The management agent

provides access to the snapshot backup repository, which contains information about the valid

snapshot backups and their relationships to the snapshot capable storage devices.

On each host an “acsgen” agent is started during the SAP cloning/ SAP system copy process.

This agent performs the storage related tasks on the source and target hosts.

http://service.sap.com/swdc

http://help.sap.com/lvmstd20

http://help.sap.com/lvment20



Figure 14 illustrates the environment:

Figure 14 Overview of the FlashCopy Manager Installation

The Tivoli FlashCopy Manager installation will follow the same steps as described in chapter

3.4.4.8 for the setup for a "Custom application environments". Additionally, a profile template

needs to be created on the management node.

The following shows a sample profile for the IBM XIV Storage:

>>> GLOBAL

# ACS_DIR /home/sapadm/acs

ACSD siccplvm 57328

# TRACE NO

<<<

>>> ACSD

ACS_REPOSITORY /home/sapadm/acs/acs_rep

# REPOSITORY_LABEL TSM

<<<

>>> DEVICE_CLASS STANDARD

COPYSERVICES_HARDWARE_TYPE XIV

# CLONE_DATABASE NO

#STORAGE_SYSTEM_ID 3984

PATH_TO_XCLI /opt/IBM/xiv/XIVGUI3.1

COPYSERVICES_SERVERNAME siccxiv

COPYSERVICES_USERNAME saplvm

BACKUP_HOST_NAME siccps98

# RECON_INTERVAL 12

# GRACE_PERIOD 24

# USE_WRITABLE_SNAPSHOTS AUTO

# USE_CONSISTENCY_GROUPS YES

<<<

>>> CLIENT

LVM_FREEZE_THAW NO

ENHANCED_PARTITIONING NO

<<<

VIOS

vFC (NPIV)

VIOS

vFC (NPIV)


SAP LVM AIX

Storage Manager native Interface e.g. IBM XIV XCLI 3.1 /opt/IBM/xiv/XIVGUI3.1/xcli

Managed Environment

IBM Tivoli FlashCopy Manager acsd /home/sapadm/acs

SAP Host Agent /usr/sap/hostctrl

SAP Source AIX

Storage Manager native Interface e.g. IBM XIV XCLI /opt/IBM/xiv/XIVGUI3.1/xcli

IBM Tivoli FlashCopy Manager acsgen /home/sapadm/acs


SAP Target AIX

Storage Manager native Interface e.g. IBM XIV XCLI /opt/IBM/xiv/XIVGUI3.1/xcli

IBM Tivoli FlashCopy Manager acsgen /home/sapadm/acs


IBM XIV SVC / Storwize Source

Target

SAN Storage • IBM XIV, IBM SVC, or IBM

Storwize V7000 Storage System

• Storage connected via vFC (NPIV)

• Compliant Storage Layout



SAP LVM takes this template and generates “runtime” templates during the actual execution.

The “ACSD” entry in the GLOBAL section of the template refers to the hostname of the

management system running the FCM acsd agent.

SAP LVM will replace the host name and port number (2nd argument) dynamically during

the actual run. IBM Tivoli Storage FlashCopy Manager traces can be enabled in this section

(set parameter “TRACE YES”).

The ACS_REPOSITORY in the ACSD section refer to the repository directory and will be

dynamically replaced too.

In the DEVICE_CLASS section, ensure that the following settings are included for IBM XIV

storage:

COPYSERVICES_HARDWARE_TYPE needs to be set to XIV

PATH_TO_XCLI needs to point to the installation directory of the XIV XCLI (see also

chapter 3.4.4.7)

COPYSERVICES_SERVERNAME needs to include the hostname / IP address of the XIV

system

COPYSERVICES_USERNAME needs to contain an administrative user on the IBM XIV

system.

IBM Tivoli Storage FlashCopy Manager uses this user to communicate with the IBM

XIV System. The user needs to be created on the IBM XIV system first, and the

password needs to be entered later during FCM setup.

In the CLIENT section, two entries need to be specified:

LVM_FREEZE_THAW NO

ENHANCED_PARTITIONING NO

Tivoli Storage FlashCopy Manager 3.2 uses IBM Global Security Kit (GSKit) for secure

communication between the distributed processes running on different LPARs. GSKit is part

of the Tivoli Storage FlashCopy Manager 3.2 software package and needs to be installed on

all LPARs.

After the configuration of Tivoli Storage FlashCopy Manager there are additional files (server

key database, client certificate) which need to be distributed across all the LPARs involved in

the scenario (central system, source system, target system).

3.3.3 LDAP Server

3.3.3.1 User Management

A SAP LVM infrastructure requires a method to synchronize naming services on all enabled

servers. This includes information on users, groups, protocols, services, hosts, etc.

Possible solutions are NIS/NIS+ or the synchronization of local files (typically stored in /etc)

using tools like rdist.

However, development of NIS has stopped and the recommended solution for centralized user

management is LDAP (Lightweight Directory Access Protocol). LDAP is widely used and

has become the standard protocol for access to information registries, optimized for read

mostly access.

AIX clients can work with any LDAP server as long as it is V3 conform. In our sample

implementation, we implemented the IBM Tivoli Directory Server V6.3 as LDAP server.



More information is available at:

http://www-01.ibm.com/software/tivoli/products/directory-server/

The following introduces some of the LDAP terminology and discusses briefly some

important steps for planning and running LDAP on AIX. This is no replacement for the

product documentation. Especially an enterprise wide usage of LDAP sharing directory

information for other services requires careful planning and you need to consider issues like

namespaces, secure authentication using SSL, and implementing high availability using server

replication. These topics are fully covered in the product documentation and the available

Redbooks and whitepapers.

3.3.3.2 Schema

A schema defines which objects and attributes can be stored in a Directory Information Tree

(DIT). It defines the names of objects and attributes, syntax, and possible values as well as

whether attributes are required or optional for an object. For example, an object could be

people or server and attributes of the objects could be name, homedirectory, uid or ip-

address, hostname.

The following schemas are available for AIX user management and authentication:

AIX The AIX schema includes the aixAccount and aixAccessGroup object classes. This

schema offers the entire AIX user and group attributes. This schema is included to support

legacy LDAP installations prior to Version 5.2.

RFC2307 The RFC2307 schema includes the posixAccount, posixGroup and other NIS

related object classes. This experimental RFC defines a schema that allows NIS maps to be

imported into LDAP. RFC2307 only defines a subset of the AIX user and group attributes.

This schema supports any RFC2307 compliant platforms and AIX 5L Version 5.2.

RFC2307AIX The RFC2307AIX schema includes the RFC2307 schema plus the AIX

specific object classes, aixAuxAccount and aixAuxGroup. The AIX specific object classes

provide attributes to store additional attributes not defined by the RFC2307 standard.

The recommended schema for AIX5.2 and higher is RFC2307AIX and has been used for the

SAP LVM reference implementation. This has several advantages such as RFC2307 is

considered a standard for UNIX systems including Linux.

3.3.3.3 Namespace

The LDAP naming model defines how entries are identified and organized. Entries are

organized in a tree-like structure called the Directory Information Tree (DIT) based on their

distinguished name (DN). A DN is a unique name that unambiguously identifies a single

entry. Distinguished Names are made up of a sequence of relative distinguished names

(RDN). Each RDN in a DN corresponds to a branch in the DIT leading from the root of the

DIT to the directory entry.

A RDN typically has the form <attribute name> = <value>. A DN is composed of a sequence

of relative distinguished names separated by commas.

The following DIT shows a sample DIT with a suffix o=isicc,c=de:

http://www-01.ibm.com/software/tivoli/products/directory-server/



Figure 15 Sample LDAP Directory Information Tree for AIX user management

3.3.3.4 LDAP Server Installation and Configuration

For LDAP server implementations please review and follow the instructions in the mentioned

references.

3.3.3.5 LDAP Client Installation

AIX LDAP client is available in a 32-bit and a 64-bit version. Their filesets are shipped with

the AIX CDs and should be included in the install images for the SAPL LVM client nodes.

3.3.3.6 LDAP Client Configuration

mksecldap is an AIX command for IBM Directory server and client setup. Please check the

man pages or the online documentation for the full syntax description.

The mksecldap command performs the required updates in various configuration files, starts

the secldapclntd daemon, and puts an entry in /etc/inittab so that the seclpdapclntd daemon

will be automatically started after reboot.

The following command is used in our demo environment to configure an AIX LDAP client:

/usr/sbin/mksecldap -c -h siccldap1,siccldap2 -a cn=root -p ldappw

There are a number of commands available to control the client daemon:

start-secldapclntd start ldap client daemon

stop-secldapclntd stop ldap client daemon

restart-secldapclntd restart the ldap client daemon

ls-secldapclntd show the status of the ldap client daemon

flush-secldapclntd clear the cache of the ldap client daemon

3.3.3.7 User Authentication

The last step required to setup user login through LDAP is to change a user’s authentication

mechanism. User authentication is controlled by the SYSTEM attribute in the

/etc/security/user file. For example to enable LDAP user c01adm to login to the local system,

you could execute the following command;

c=de

o=isicc

cn=user

uid=c01adm

uid=orac01

cn=group

gid=sapsys

gid=dba



chuser SYSTEM=LDAP registry=LDAP c01adm

To allow all LDAP users to login to the system, one can set each LDAP user's SYSTEM

attribute to LDAP, or simply set the default stanza's SYSTEM attribute to LDAP. This has to

be done manually in the /etc/security/user file, where you need to change the following two

attributes in the default stanza.

Default:

SYSTEM = "LDAP or compat"

registry = LDAP

We recommend using local authentication for the root user so that you can still login to the

server in case there is something wrong with the LDAP setup. With local authentication the

root user’s stanza should look like:

root:

SYSTEM = "compat"

registry = files

In our sample environment the complete client setup is automated in a NIM script. The script

uses mksecldap to configure the client and then runs a Perl command to automatically change

the default and root user’s stanza to the appropriate:

Sample Customizing Script for Automated LDAP Client Setup #Setup LDAP Client

#AIX users are managed by LDAP

/usr/sbin/mksecldap -c -h siccldap1,siccldap2 -a cn=root -p ldappw

perl -p -i -e 'if ( /default:/ .. /SYSTEM =/ ) {

s/"compat"/"LDAP or compat"\n\tregistry = LDAP/;

} elsif ( /root:/ .. /SYSTEM =/ ) {

s/"compat"/"compat"\n\tregistry = files/;

}' /etc/security/user

3.3.3.8 Create Users and Groups

The AIX mkuser command is able to create a user on the LDAP server using the load module

option “-R LDAP”. Unfortunately the “smit mkuser” panel doesn’t allow for this option. You

should create a new user using the mkuser command. Once a user is created, you can use the

“smit chuser” panel to change user attributes. For example to create a user c01adm, use the

command

mkuser –R LDAP registry=LDAP c01adm

Then use the “smit chuser” panel to change the other required attributes (of course, you could

also set them directly using the appropriate attribute options with the mkuser command).

The problem also applies to the mkgroup command, so you have to create the required groups

manually as well:

mkgroup –R LDAP sapsys

Consider using some naming conventions for the numerical user id, for example by reserving

certain ranges for the SAP and database administration users:



mkgroup -R LDAP registry=LDAP id=400 dbqa1adm

mkgroup -R LDAP registry=LDAP id=401 dbqa1ctl

mkgroup -R LDAP registry=LDAP id=402 dbqa1mnt

mkgroup -R LDAP registry=LDAP id=403 dbqa1mon

mkgroup -R LDAP registry=LDAP id=404 sapsys

mkgroup -R LDAP registry=LDAP id=405 sapinst

mkuser -R LDAP registry=LDAP id=400 pgrp=sapsys

groups='sapsys,dbqa1ctl,dbqa1mon,sapinst' home='/home/qa1adm' gecos='SAP System

Administrator' shell='/usr/bin/csh' fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151

qa1adm

mkuser -R LDAP registry=LDAP id=401 pgrp=dbqa1adm groups='dbqa1adm'

home='/db2/db2qa1' gecos='SAP Database Administrator' shell='/usr/bin/csh'

fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151 db2qa1

mkuser -R LDAP registry=LDAP id=402 pgrp=dbqa1mnt groups='dbqa1mnt'

home='/home/sapqa1' gecos='ABAP Database Connect User' shell='/usr/bin/csh'

fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151 sapqa1

mkuser -R LDAP registry=LDAP id=403 pgrp=dbqa1mon groups='dbqa1mon'

home='/home/sapqa1db' gecos='Java Database Connect User' shell='/usr/bin/csh'

fsize=-1 cpu=-1 data=-1 stack=-1 core=2097151 sapqa1db

3.3.4 Other Central Management Components

3.3.4.1 OS Management

We highly recommend a centralized OS software management concept.

There are two common solutions to manage the installation and maintenance of AIX images

in a server landscape. The first one is using the AIX built-in Network Installation

Management (NIM) features. The second one is to use the VMControl plugin of IBM Flex

System Manager or IBM Systems Director.

NIM allows the installation and maintenance of AIX base operating system and additional

software from a central repository. As images are sent across the network to the partition to be

installed, the installation steps can be performed fully remotely without needing physical

access to the server. During and after base installation, you can customize the machines by

running specific customizing scripts. Later on, you can deploy fixes and new software images

to the server partitions from a central administration point.

NIM supports several different machine types: diskless, dataless and standalone. A diskless

machine has no own disks, all file systems and other resources are installed remotely and

accessed via the network. A dataless machine has local paging and dump devices, the file

systems for / and /usr are located on a remote server, all the other file systems are located

either local or remote. On a standalone machine, all file systems and resources are installed

on local disks. Due to performance and availability reasons, we recommend using local (or

SAN attached disks) on the computing nodes and install them as standalone machines. It is

possible to use diskless or dataless machines for test and development systems. They boot of

the network and mount the OS from the NIM server.

The following two Redbooks describe in detail the concepts of centralized OS management

using NIM or VMcontrol:

NIM from A to Z in AIX 5L

http://www.redbooks.ibm.com/abstracts/sg247296.html?Open

IBM Systems Director VMControl Implementation Guide on IBM Power Systems






3.3.4.2 Network Infrastructure

There are no special network infrastructure requirements for an SAP LVM landscape.

Common standard today and recommended environment is a switched Gigabit Ethernet

network infrastructure.

The major difference compared to a traditional SAP implementation is the consistent usage of

virtual IP names and addresses assigned to a SAP instance. These addresses need to be

implemented in the customer’s name services infrastructure, typically based on DNS (please

refer to the section 3.1. for some recommendations about naming conventions for virtual IP

names).

If virtual network addresses come from new subnets, one might have to update router tables,

saprouter definitions, etc. to ensure that the end user can access the SAP instance using the

virtual IP name. You also have to change SAP logon groups to reflect the virtual IP names.

Note 962955 - Use of virtual TCP/IP host names

3.4 Install System Landscape

The following section provides an overview about the preparation steps required for the

environment. This includes considerations about

PowerVM and VIOS

IBM Flex System Manager or IBM Systems Director

IBM HMC

OS setup for the hosts

SAP Installation

3.4.1 PowerVM and VIOS

Base assumption is that the volume(s) for the AIX operating system (rootvg) is/are located on

the same storage system as the SAP system, and that the “rootvg” storage volumes are also

attached via virtual fibre channel (vFC). In this case, the “host” and “adapter” definition in the

storage system required for the access to the “rootvg” volumes will ensure the required

connectivity to the SAP application related volumes in the storage system also.

3.4.2 IBM Systems Director Setup

The following describes high-level configuration steps required for the integration of either

IBM Systems Director or IBM Flex System Manager with SAP LVM. Further details are

described in SAP Note 1728222 - Configuration of IBM Systems Director VMControl for

SAP LVM

The term “virtualization manager” is applied for both integration scenarios. IBM Systems

Director and IBM Flex System Manager both require the identical steps for the setup. The

tasks on the virtualization manager include:

Discover the IBM HMCs on the virtualization manager and configure access to them.

Perform a "collect inventory" task on the virtualization manager for IBM HMC, Power

servers, and for all logical partitions (LPARs).






Discover operating systems running on partitions, configure access, and perform

"collect inventory" task on virtualization manager for all the discovered operating

systems.

Define a user for SAP LVM access on the virtualization manager

The default administrator user can be used as user for SAP LVM. The administrators

in SAP LVM then have the same credentials and get data for all the IBM Power

Systems configured in the virtualization manager.

To restrict the access to the specific set of IBM Power Systems managed by SAP

LVM, a specific user can be created in the virtualization manager. This user does not

have the credentials for all Power systems available in the virtualization manager, but

only to those in scope of the SAP LVM environment. The configuration is described

in chapter 4.9.

Create a security certificate on the virtualization manager

Per default, the virtualization manager uses Port 8422 for communication.

Communication via this port must be allowed in the network and firewall settings for

the environment.

o Open the URL “https://<virtualization manager host>:8422" in a browser

window. During the first connection attempt the browser displays a notification

window to accept the certificate. This virtualization manager certificate is

stored in the internet browser configuration locally, and can be exported to a

certificate file.

o The virtualization manager certificate needs to be imported into the SAP

NetWeaver system with the deployed SAP LVM software.

The certificate store must be "TrustedCAs".

Check that the hostname in the certificate is valid.

For more information on this topic see also the SAP documentation "Using the

AS Java Key Storage".

Using the virtualization manager for AIX OS provisioning requires to discover, to

access and to perform "collect inventory" tasks for NIM, VIO servers, fibre channel

switches and storage managers in addition.

It must be possible to

Capture and deploy an AIX operating system via NIM (4.9)

Select a particular IBM Power server as target for deployment

Select a storage system for deployment

within the different layers of the virtualization manager (VMControl, Storage Manager

or Control). Then similar provisioning operations can be triggered out of SAP LVM.

Using the virtualization manager for LPAR relocation (Live Partition Mobility)

requires the configuration of “server pools” or “virtual farms”. If it is intended to

migrate LPAR "X" hosted on IBM Power system "A" to IBM Power system "B", then

http://help.sap.com/saphelp_nwce10/helpdata/en/e9/a1dd44d2c83c43afb5ec8a4292f3e0/content.htm

http://help.sap.com/saphelp_nwce10/helpdata/en/e9/a1dd44d2c83c43afb5ec8a4292f3e0/content.htm



both Power system "A" and Power system "B" must be members of the same server

pool or virtual farm. If the virtualization manager is able to relocate a partition, the

same operation can be triggered in SAP LVM too.

3.4.3 HMC Setup

The recommended setup for SAP LVM is to use IBM Flex System Manager or IBM Systems

Director as virtualization manager. However, the management via IBM Power HMC is

possible too.

This is described in detail in SAP Note 1728293 - Configuration of IBM Power HMC for

SAP LVM.

3.4.4 Install Hosts

This section describes the base configuration steps on an AIX operating system level for the

compute nodes. All the required steps need to be executed at least on the first system image.

This image can then be captured (NIM, VMControl) and can be used as a "master image”

template for the deployment of all additional compute nodes.

The required steps include

Activate IPsec

Check/ update DNS configuration

Configure the LDAP client

Adapt /etc/services base configuration

Check/adapt filesystem sizes

Install SAP Host Agent package and SAPACEXT libraries

Install XIVGUI (IBM XIV Storage systems)

Configure ssh (IBM SVC or Storwize systems)

Install IBM Tivoli Storage FlashCopy Manager

3.4.4.1 Activate IPsec

Each of the hosts participating in a system provisioning use case for SAP system cloning or

SAP system copy needs to be prepared for network isolation.

In an AIX environment, the network isolation is provided via IPsec. IP security, commonly

known as IPsec, is a protocol developed by the Internet Engineering Task Force (IETF),

designed to provide "end-to-end" cryptographically based security for IP network

connections. IPsec implementations include a method of restricting connections to various

services, based on their origin and destination.

This feature is known as packet filtering. All packets on an IP network originate from an IP

address and a port, and are destined for another IP address and port. A packet filter is a

physical or virtual device that sits between the endpoints of a connection and determines

whether the packet should be permitted to continue to its destination. The decision is made by

comparing various attributes of the packet to rules that are defined by the administrator of the

packet filter. Those attributes include source address and subnet, source port, destination

address and subnet, destination port, protocol, direction of the connection, and fragmentation

of the packet.

Make sure that APAR IZ98741 "DYNAMIC FILTER RULE ID NOT MODIFIED BY

RMFILT -N ALL" is installed in the AIX OS image

IPsec is activated via the AIX SMIT tool using fast path “ips4_start”:





# smitty ips4_start

Figure 16 Start IPsec

Alternatively, IPsec can be started directly via the command line:

# /usr/sbin/mkdev -c ipsec -t 4

No manual definitions are required for packet filters. SAP LVM defines or releases

automatically the packet filters for system isolation during the execution of the System Copy

or System Clone workflows.

Figure 17 illustrates an additionally required step during the definition of a managed host. The

checkbox “Isolation Ready” must be enabled so that the LPAR can participate as a target host

in a System Copy/System Clone scenario:

Figure 17 Define the Host as “Isolation Ready” in SAP LVM

3.4.4.2 DNS Name Resolution

Ensure that the search order for DNS name resolution is first based on the local /etc/hosts file,

and then on the external DNS nameserver.



# vi /etc/netsvc.conf

hosts = local,bind4

In our test environment, the content of the /etc/hosts file contains only the IP label for the

hostname of the partition. All external addresses are resolved via DNS. In this case,

communication to the DNS server needs to be allowed during the “isolated”, “network

fenced” state within the System Clone / System Copy workflow. The required settings to

allow DNS communication are illustrated in chapter 4.7, Figure 42.

3.4.4.3 Configure LDAP Client

As prerequisite for the use of LDAP the IBM Tivoli Directory Server Client needs to be

installed on each LPAR.

idsldap.clt32bit61

idsldap.clt64bit61

The LDAP client needs to be defined on the LPAR in the same way as described in chapter

3.3.3.5 and 3.3.3.6.

3.4.4.4 Adapt /etc/services Entries

Table 8 illustrates the ports to be predefined for ABAP instances. For simplicity all ports

covering the full instance number range from 00 to 99 are added to /etc/services.

In total 400 ports for sapdpXX, sapgwXX, sapdpXXs and sapgwXXs are added. If only a

subset of the ports is required in the organization (e.g. due to SAP instance number standards)

then it is sufficient to add only those specific ports to the /etc/services file.

Port Description /etc/services content

sapdpXX

XX { 00, 01, 02, … 99 }

SAP System Dispatcher Ports sapdp00 3200/tcp

…

sapdp99 3299/tcp

sapgwXX

XX { 00, 01, 02, … 99 }

SAP System Gateway Ports sapgw00 3300/tcp

…

sapgw99 3399/tcp

sapdpXXs

XX { 00, 01, 02, … 99 }

SAP System Dispatcher Security Ports sapdp00s 4700/tcp

…

sapdp99s 4799/tcp

sapgwXXs

XX { 00, 01, 02, … 99 }

SAP System Gateway Security Port sapgw00s 4800/tcp

…

sapgw99s 4899/tcp

Table 8 Entries in /etc/services

3.4.4.5 Adapt File System Sizes

Several software components need to be available on each of the LPARs in the managed

environment. If all the software components are installed locally on the LPAR, then sufficient

free space is required in certain filesystems for the management components. Table 9 gives an

example about required sizes for the components in the “local” filesystems/directories:

Directory Size

(approx.)

MByte

VG Content

/usr/sap/hostctrl 200 rootvg or saplocalvg

SAP Host Agent package, together with the SAPACEXT libraries

/opt/IBM/xiv/XIVGUI3.1 (example)

350 rootvg XIV XCLI Installation



Directory Size (approx.)

MByte

VG Content

/usr/tivoli/tsfcm 700 rootvg Installation directory for IBM Tivoli Storage

/home/sapadm/acs (example) 700 rootvg or saplocalvg

Work directory for IBM Tivoli Storage FlashCopy Manager (SAP LVM context)

Table 9 Local Filesystem layout

Increase the size of the /usr filesystem to allow to install the SAP Host Agent package

and the IBM Tivoli Storage FlashCopy Manager software (FCM)

Increase the size of the /home filesystem for the copy of the FCM package in

/home/sapadm/acs

Increase the size of the /opt filesystem for the XIVGUI package

# chfs -a size=+1500M /usr

# chfs –a size=+1G /home

# chfs -a size=+1G /opt



3.4.4.6 Install SAP Host Agent and SAP Adaptive Computing Extensions

The runtime user for the SAP Host Agent is user “sapadm”. SAP LVM uses this user id to

authenticate with the SAP Host Agent on the managed system. The user “sapadm” must be

available on a compute node before the SAP Host Agent installation.

The SAP Host Agent package requires “root” user credentials during installation.

The group “sapsys” must exist on the host.

The user “sapadm” must exist and be a member of the “sapsys” group.

Create a home directory for sapadm. The FCM setup on the managed LPAR is

performed in the home directory of user sapadm

# mkdir /home/sapadm

# chown sapadm.sapsys /home/sapadm

Download the recent SAPHOSTAGENT.SAR archive and unpack it using the

SAPCAR tool.

The archive contains the “saphostexec” program. The SAP Host Agent package is

installed/upgraded by executing “saphostexec –install”, respectively “saphostexec –

upgrade” as root user. This installs the SAP Host Agent package in directory

/usr/sap/hostctrl.

Some SAP Host Agent runtime executables have the setuid bit set allowing for the

execution of privileged commands during SAP LVM prepare and unprepare

operations.

Download the SAPACEXT.SAR archive and unpack it to the operations.d

subdirectory of the SAP Host Agent installation.

# cd /usr/sap/hostctrl/exe/operations.d

# SAPCAR –xvf /tmp/SAPACEXT.SAR

Additional considerations for the SAP Host Agent are found in the SAP Note

Note 1292836 - Logfiles of the SAP Host Agent

3.4.4.7 Install XIVGUI

In the proof of concept environment XIVGUI 3.1 build 18 is installed. The XIVGUI image is

deployed to the directory /opt/IBM/xiv (XIVGUI is part of the OS image).

Parameter Value

XCLI Installation Directory /opt/IBM/xiv/XIVGUI3.1

Table 10 XIVGUI Install directory

At first create the target directories for the XIVGUI image

# mkdir –p /opt/IBM/xiv/XIVGUI3.1




Install additional rpm packages for the XIVGUI image.

Although this is a prerequisite this is not explicitly mentioned in the XIV XCLI

installation guide: Required rpm packages include:

o libgcc

o libstdc

o readline-5

readline-5 additionally requires

o bash

o info

# rpm -i libgcc-4.2.0-3.aix6.1.ppc.rpm

# rpm -i libstdc++-4.2.0-3.aix6.1.ppc.rpm

# rpm -i gcc-4.2.0-3.aix6.1.ppc.rpm

# rpm -i bash-4.2-6.aix5.1.ppc.rpm

# rpm -i info-4.6-1.aix5.1.ppc.rpm

# rpm -i readline-5.2-3.aix5.1.ppc.rpm

Adjust the library settings and the LIBPATH to include these shared libraries

# cd /opt/freeware/lib

# ln -s /opt/freeware/lib/gcc/powerpc-ibm-aix6.1.0.0/4.2.0/libgcc_s.a

# ln -s /opt/freeware/lib/gcc/powerpc-ibm-

aix6.1.0.0/4.2.0/libstdc++.a

# export LIBPATH=$LIBPATH:/opt/freeware/lib

Extract the XIVGUI software from the Software Image Repository

# ./xivgui-3.1-build18-aix.bin

Adapt the xcli script

There is a bug in the xcli script: xcli can be started only out of the installation path.

xcli needs to be patched to be able to start it from any location.

Edit the xcli script and add the line

prg="$0"

in the upper section of the xcli script

Validate the XCLI by a test connection to the IBM XIV storage system

# /opt/IBM/xiv/XIVGUI3.1/xcli

User Name: ...

Password: ...

Machine IP/Hostname: ...

connecting.

XIV 1300360>>



3.4.4.8 Install IBM Tivoli Storage FlashCopy Manager

SAP LVM exploits IBM Tivoli FlashCopy Manager for the backup of "Custom application

environments". On the managed systems, the “acsgen” agent is used to invoke the different

storage related activities on the LPAR during the System Copy/ System Cloning process.

Base Installation of IBM Tivoli Storage FlashCopy Manager

./3.2.0.1-TIV-TSFCMFTP-AIX.bin

Parameters:

- Choose Locale: 2- English

- PRESS <ENTER> TO CONTINUE

- Press Enter to continue viewing the license agreement, or enter "1" to

accept the agreement

- Choose the Install Set to be installed by this installer: 4 - IBM

Tivoli Storage FlashCopy (R) Manager (Custom Applications)

- ENTER AN ABSOLUTE PATH: /usr/tivoli/tsfcm/acs_3.2.0.1

- IS THIS CORRECT? (Y/N): Y

- PRESS <ENTER> TO CONTINUE

- PRESS <ENTER> TO EXIT THE INSTALLER

Check the license file (contained in IBM Tivoli FlashCopy Manager base installation).

If the FCM 3.2.0.1 package was downloaded as an update package, it will not include

the license file. The license file included in the FCM 3.2.0.0 base package needs to be

copied to the directory:

o Copy license file “tsmacs.lic” from the FCM 3.2.0.0 base package to the FCM

install directory /usr/tivoli/tsfcm/acs_3.2.0.1

Create Directory for FCM profile / environment:

# mkdir /home/sapadm

# chown sapadm:sapsys /home/sapadm

# mkdir /home/sapadm/acs

# chown sapadm:sapsys /home/sapadm/acs

Run the setup for FCM custom application

# cd /usr/tivoli/tsfcm/acs_3.2.0.1

# /setup_gen.sh

Script not started from 'acs' sub-directory of the Home directory of

a custom application user. Trying to locate the correct Home

directory or install IBM Tivoli Storage FlashCopy(R) Manager in a new

Home directory ...

Please enter the Home directory:

/home/sapadm



This is a new installation of IBM Tivoli Storage FlashCopy(R) Manager

into this Home directory.

Please enter the name of the user of this Home directory:

sapadm

Please enter the group name of the user:

sapsys

Installing/Updating IBM Tivoli Storage FlashCopy(R) Manager

installation for /home/sapadm/acs

Checking environment

- - - - - - - - - - -

+ OK: installation directory exists.

+ OK: setup_gen.sh runs in IBM Tivoli Storage FlashCopy(R) Manager

installation directory '/usr/tivoli/tsfcm/acs_3.2.0.1'

+ OK: IBM Tivoli Storage FlashCopy(R) Manager installation will be

installed to '/home/sapadm' for the first time.

Installation messages

- - - - - - - - - - -

+ OK: New installation !

+ OK: IBM Tivoli Storage FlashCopy(R) Manager directory

'/home/sapadm/acs' created successfully.

+ OK: Files successfully copied.

checking /home/sapadm/acs/acsgen ...

OK

Select one of these configurations:

(1) On-Site Production System configuration with optional remote

Backup System configuration

(2) On-Site Backup System configuration

Enter '1' to configure IBM Tivoli Storage FlashCopy(R) Manager on the

production server with the option to remotely synchronize the

configuration of one or multiple backup systems using Secure Shell.

Enter '2' to configure IBM Tivoli Storage FlashCopy(R) Manager on the

backup system as a 'separate installation'.

2

Going to install on backup server



Please enter the directory where the profile will be stored:

[/home/sapadm/acs]

Deactivating IBM Tivoli Storage FlashCopy(R) Manager management

daemon.

Are you going to perform off-loaded backups to Tivoli Storage

Manager? [y|n]

n

Profile configuration

=====================

Special commands:

'?' ==> Show help for current parameter

'!d' ==> Delete current parameter instance (only applicable

to multi-instance parameters)

FMM8422I Creating new profile '/home/sapadm/acs/profile' for

application 'GENERIC' ...

****** Profile parameters for section GLOBAL : ******

Path of the IBM Tivoli Storage FlashCopy(R) Manager directory

{ACS_DIR} = [/home/sapadm/acs]

Hostname and port of machine running Management Agent {ACSD}

(<hostname> <port>) = [] siccplvm 57328

Low level tracing {TRACE} (YES|NO) = [NO]

FMM8426I Saving profile '/home/sapadm/acs/profile' ...

FMM1555I Profile successfully created.

Please enter the password for authentication with the ACS daemon:

Please re-enter password for verification:

Creating password file at /home/sapadm/acs/shared/pwd.acsd.

A copy of this file needs to be available to all components that

connect to acsd.

Enter the device classes to use for this Backup System. Make sure

each device class is separated by a comma.



STANDARD

IBM Tivoli Storage FlashCopy(R) Manager requires at least two daemon

processes to be running at all times.

IBM Tivoli Storage FlashCopy(R) Manager can add the necessary daemon

processes to the inittab.

Alternatively you may choose to start and stop these processes

yourself, for instance if you want to include the processes in your

HA policy.

Do you want IBM Tivoli Storage FlashCopy(R) Manager to create the

inittab entries for you? [y|n]

n

Please run the following daemon processes:

/home/sapadm/acs/acsgen -D -M -s STANDARD

Please make sure that the command '/home/sapadm/acs/tsm4acs' is

started by a scheduler or manually.

No special customization for the FCM profile is required on the host. SAP LVM will

derive the required content for the profile out of the master profile on the FCM

management LPAR (“acsd”), and copy the profile to the source and target hosts during

the execution of the SAP System Cloning/SAP System Copy workflow.

No special customization for the FCM profile is required on the host.

Install Global Security Kit

IBM Global Security Kit (GSKit) needs to be installed on all LPARs participating in

the scenario. GSKit can be installed using the setup script:

# cd /home/sapadm/acs

# ./setup_gen.sh -a install_gskit -d /home/sapadm

Afterwards, check if the filesets are properly installed:

lslpp -l GSKit8.gskcrypt64.ppc.rte

lslpp -l GSKit8.gskssl64.ppc.rte

Distribute certificate files

Distribute the files for the Server Key DB

o fcmcert.crl

o fcmcert.rdb

o fcmcert.kdb

o fcmcert.sth

and the Client Certificate

o fcmselfcert.arm

across all LPARs (directory /home/sapadm/acs in our case).

Tivoli Storage FlashCopy Manager password file

The FlashCopy Manager configuration on the central node (acsd) generates a

password file in /home/sapadm/acs/shared/pwd.acsd. The password file needs to be

copied to all the FlashCopy Manager directories on the target hosts. It contains the



connection password to the acsd, and also the connection password to the Storage

System.

A NFS-shared directory for the “ACS” directory would simplify the setup for the

password file and the certificates, but was not realized in the lab environment.

3.4.5 SAP System

In the following section, base considerations for the installation of a new SAP system, as well

as the eventually required modifications of an existing SAP system are given. These

considerations need to be taken into account, before the SAP system then can be registered

with SAP LVM. The definition of the new system in the SAP LVM context is described in

chapter 4.7

3.4.5.1 Filesystem considerations for the SAP Systems (SAP System Copy)

3.4.5.1.1 DB2 UDB LUW

The two tables below illustrate two different storage layouts for a SAP System based on DB2

UDB. The first example (Table 11) includes a simple layout, having a few different

filesystems only and just separating between SAP application and database filesystems. The

filesystem layout fulfils the requirements for the relocation and SAP system copy scenarios

with SAP LVM.

Place all DB2 UDB Files into one common AIX Volume Group for the database: Filesystem Volume Group Content

/db2 vg01 Contains the instance directory /db2/db2<sid> and all other database related directories. All DB2 database specific data is stored in a directory structure below the filesystem /db2

/var/db2 vg01 Filesystem for DB2 global registry

/home/<sid>adm vg00 Filesystem for home directory of SAP admin user

/sapmnt/<SID> vg00 Filesystem for SAP profile, SAP work/interface data, and SAP Kernel source

/usr/sap/<SID> vg00 Filesystem for SAP Instance directories

/usr/sap/DAA/SMDA97 vg00 Filesystem for SMD agent

Table 11 Minimum filesystem layout (DB2 UDB)

In the second example (Table 12) the filesystem layout is more complex. The database layout

is split-up into different volume groups. The filesystem layout fulfils the requirements for the

relocation and SAP system copy scenarios with SAP LVM as well as the capability for

performing SnapShot/FlashCopy backups for the database with IBM Tivoli Storage

FlashCopy manager.

Separate DB2 datafiles, active logfiles, and all other DB2 files into three different AIX

Volume Groups

Make sure that the database directory /db2/<SID>/db2<sid> is part of the volume

group with the data files



Filesystem Volume Group Content

/db2/ vg01 Contains the instance directory /db2/db2<sid>.

/db2/<SID> vg02 Contains the database directory. In case that regular SnapShot backups are intended for the database, the content needs to be included in the SnapShot image together with the DB2 database containers/ storage paths

/db2/<SID>/log_archive vg01 Separate filesystem for DB2 inactive logs

/db2/<SID>/log_dir vg03 Separate filesystem for DB2 active logs: In case SnapShot backups are regularly taken and should be used for a fast restore, this filesystem *must* be separate from database data.

/db2/<SID>/db2dump vg01 Separate filesystem for the db2diag.log for convenience

/db2/<SID>/sapdataX vg02 Contains the database containers / storage paths





/var/db2 vg01 Filesystem for DB2 global registry

Table 12 Extended filesystem layout (DB2 UDB)

If FlashCopy backups for the database are a requirement, then a layout similar to the second

example has to be implemented.

3.4.5.1.2 Oracle Database

The two tables below illustrate two different storage layouts for a SAP System based on an

Oracle database. The first example (Table 11) includes a simple layout, having a few different

filesystems only and just separating between SAP application and database filesystems. The

filesystem layout fulfils the requirements for relocation and SAP system copy scenarios with

SAP LVM.

Place all ORACLE Files into one common AIX Volume Group for the database: Filesystem Volume Group Content

/oracle vg01 Contains all database related directories.





Table 13 Minimum filesystem layout (Oracle)



In the second example (Table 12) the filesystem layout is more complex. The database layout

is split-up into different volume groups. The filesystem layout fulfils the requirements for

relocation and SAP system copy scenarios as well as the capability for performing

SnapShot/FlashCopy backups for the database with IBM Tivoli Storage FlashCopy manager:

Separate Oracle datafiles, online logfiles and other Oracle files into three different

AIX volume groups

Relocate all „non“ datafiles (e.g. Oracle control files) from the AIX volume group for

the Oracle datafiles to a new location

o The SAP DB installation per default stores one control file below

/oracle/<SID>/sapdata1

Example for „new“ control file locations:

/oracle/<SID>/origlogA/cntrl/cntrl<SID>.dbf

/oracle/<SID>/origlogB/cntrl/cntrl<SID>.dbf

/oracle/<SID>/mirrlogA/cntrl/cntrl<SID>.dbf

An additional, new filesystem is required for the redo-logs written during execution of

a SAP System Copy or Clone (for the rollforward recovery of the database). This

filesystem is either exported via NFS and mounted on the target host, or is also Flash-

copied and mounted on the target.

either exported via NFS and gets mounted on the target host, or is also Flash-copied

and mounted on the target.

Filesystem Volume Group Content

vg01 Oracle binaries

/oracle/<SID> vg02 Contains the database directory. In case that regular SnapShot backups are intended for the database, the content needs to be included in the SnapShot image together with the Oracle data files

/oracle/<SID>/oraarch/ vg01 Separate filesystem for Oracle archive logs

/oracle/<SID>/origlogA /oracle/<SID>/origlogB /oracle/<SID>/mirrlogA /oracle/<SID>/mirrlogB

vg03 Separate filesystem for DB2 online logs: In case SnapShot backups are regularly taken and should be used for a fast restore, this filesystem *must* be separated from database data.

/oracle/<SID>/saptrace vg01

/oracle/<SID>/sapreorg vg01

/oracle/<SID>/sapdataX vg02 Contains the database containers/storage paths





Table 14 Extended filesystem layout (Oracle)



If FlashCopy backups for the database are a requirement, then a layout similar to the second

example has to be implemented.

3.4.5.1.2.1 Filesystem for Oracle Redo Logs for System Clone and System Copy

During the SAP LVM Step “Clone Volumes with DB backup mode” (execution of the storage

based copy), the Oracle database is set into “begin backup” mode. For the duration of this

mode data of changed database blocks (with full “before” and “after” information) is written

to the Oracle database logs.

This redo log information is required on the target system to start the database clone.

When the storage based copy is complete, SAP LVM copies

an Oracle DB configuration file

a backup of the Oracle control file

all (archived) logs written during the time frame “BEGIN BACKUP” until “END

BACKUP”

to the filesystem for Oracle Redo Logs.

The filesystem is then accessed on the target system for the Oracle rollforward recovery to the

last committed transaction of the source system.

The filesystem is part of the SAP system configuration in LVM:

Figure 18 Filesystem for Oracle Redo Logs

If the filesystem is shared via NFS, the following settings need to be maintained:

Usage Type Oracle Redo Logs for Online Clone

Storage Type NETFS

Mount Point (1) JFS2 filesystem of „cross-mount“

(2) n/a (but needs to be specified, e.g. /tmp/VCM)

Export Path NFS exported fs (<NFS Server>:<NFS export> )

Mount Options hard mount, background, interrupt, read and write size 32768

(The settings are Oracle requirements for a BACKUP CONTROLFILE

command to a NFS-mounted filesystem)

In case of NFS sharing, SAP LVM will mount the exported directory to

/tmp/VCM/<TGT_SID> on the hosts.

A stanza for /tmp/VCM/<TGT_SID> is required (with all mount options) in

/etc/filesystems

The NFS export needs to allow „root“ access for both the source and target host (no

„root squash“)

If multiple NFS client addresses (e.g. virtual IP addresses of DB and CI) are present in

the sub-network for the NFS server access, then all of them need to be granted access

in the export list on the NFS server



3.4.5.2 Install a new SAP system

For the SAP system installation, select one host of the SAP LVM environment that has no

active SAP instances.

The filesystem layout needs to be compliant to chapter 3.4.5.1

For the SAP system installation the virtual IP addresses for the database instance and

the central instance/central services need to be active. Add the virtual IP addresses to

the network adapter with the “ifconfig” command. These addresses must not be

defined as persistent, as the virtual IP addresses for the SAP instances later on will be

managed exclusively by SAP LVM during the prepare/unprepare workflows.

The ifconfig syntax to define an alias on an existing interface is:

# ifconfig <interface> alias <virtual IP> netmask <netmask>

For example: # ifconfig en0 alias c01scs netmask 255.255.255.0

# ifconfig en0 alias c01dbs netmask 255.255.255.0

During the SAP system installation, use the SAPINST_USE_HOSTNAME

parameter to specify the virtual hostname for the instance to be installed.

Users and Groups should be defined centrally in the LDAP environment. As SAPinst

or Software Provisioning Manager (SWPM) currently cannot handle LDAP users

properly, create users and groups for SAP and DB administration manually before the

SAP system installation.

The detailed installation procedure needs to follow all the steps described in the SAP

installation documentation and the respective SAP notes.

3.4.5.3 Adapt an already existing SAP system

File System Layout

The filesystem layout needs to be compliant to chapter 3.4.5.1. It may be necessary to

copy or move the content of the standard installation directories into a SAP LVM

compliant storage layout.

Virtual IP Address and Hostname

Within the SAP LVM environment the database server and the central instance/central

services can be operated on separate hosts. Two different virtual IP addresses are

required for them. If one of the virtual IP addresses was defined as base address of a

network adapter, redefine the network adapter to a new address. Define the virtual IP

addresses as IP alias to the network adapter using the ifconfig command:

If the hostname for the SAP instance is changed to a new virtual hostname, make sure

to adapt all occurrences of the name in the filenames and contents of DEFAULT,

START, and instance profiles. Create and update the following parameters in the

instance profile to the virtual hostname too.

SAPLOCALHOST = <virtual hostname CI>

SAPLOCALHOSTFULL = <full qualified virtual hostname CI>



icm/host_name_full = <full qualified virtual hostname CI>

For example, in the default profile DEFAULT.PFL, the SAPDBHOST has to reflect

the virtual IP label of the DB server, and the virtual IP label of the CI will be part of

the mshost, vbname, enqname, btcname, and gateway parameters:

SAPDBHOST = c01dbs

rdisp/mshost = c01scs

rdisp/vbname = c01pas_C01_10

rdisp/enqname = c01pas_C01_10

rdisp/btcname = c01pas_C01_10

Dependent on your individual environment, you may have to adapt settings in further

interfaces/SAP systems too.

(Re-) create the Users in LDAP

The administrative users for Database and SAP system have to be created with the

same numeric UID in the LDAP directory. If the UID for the users violate uniqueness

within the SAP LVM environment, the ownership of all files needs to be changed to

an unique UID.

Adapt user environment files

During shell startup, several files are sourced to create the proper environment for

SAP admin and DB admin users. During installation time, these files are created

containing the hostname within their name. Rename them in the home directory of

both SAP admin and DB admin user.

# mv .sapenv_<virtual hostname>.sh .sapenv.sh

# mv .apoenv_<virtual hostname>.sh .apoenv.sh

# mv .dbenv_<virtual hostname>.sh .dbenv.sh

# mv .sapenv_<virtual hostname>.csh .sapenv.csh

# mv .apoenv_<virtual hostname>.csh .apoenv.csh

# mv .dbenv_<virtual hostname>.csh .dbenv.csh

You also need to check the shell start profiles (.cshrc) that it sources all required

environment files correctly. Each entry should look like

# RDBMS environment

if ( -e $HOME/.dbenv_`hostname`.csh ) then

source $HOME/.dbenv_`hostname`.csh

else if ( -e $HOME/.dbenv.csh ) then

source $HOME/.dbenv.csh

endif

This means the shell first searches for a file containing the hostname, will not find it

and then uses the corresponding file without the hostname. Sometimes the “else if”

clause is missing and you need to create it manually.

Adapt database configuration files to virtual IP address

o ORACLE: Adapt configuration files tnsnames.ora and listener.ora (ADDRESS =

(COMMUNITY = SAP.WORLD)

(PROTOCOL = TCP)



(HOST = c01dbs)

(PORT = 1537)

)

The stanzas in tnsnames.ora and listener.ora have to reflect the virtual

hostname of the DB instance. The communication port should be unique within

the environment.

o DB2 UDB: Update db2cli.ini to the virtual DB server address

; Comment lines start with a semi-colon.

[C01]

Database=C01

Protocol=tcpip

Hostname=c01dbs

Servicename=5912

[COMMON]

Diagpath=/usr/sap/C01/SYS/global/db6/db2dump

The “Hostname” in the db2cli.ini file needs to reflect the virtual hostname of

the DB instance. The service name reflects the communication port and should

be unique within the environment.

Perform a test connection to the database: Start the database (and, in case of DB

ORACLE, the listener process). As user <sid>adm, perform a test connection to the

database using R3trans:

c01adm> R3trans –d –v

This is R3trans version 6.22 (release 720 - 03.06.11 - 17:57:00).

unicode enabled version

R3trans finished (0000).

R3trans should return 0 for a successful DB connection.

3.4.5.4 Additional SAP profile parameters

See also SAP Note 1438774 - New profile parameter system/uuid and system/description:

SAP LVM will verify the system/uuid profile parameter against its configuration prior to

any operation (start, stop, relocate) on any instance of a (cloned) system, to ensure that the

operation is triggered on the correct instances. The system/description profile parameter

contains a verbal description of the system.

The parameter can be added to the SAP system later on also during the LVM configuration

phase. However, the SAP system then needs to be restarted to activate the parameter.

3.4.5.5 SAP license considerations

The license key for an SAP system is checked on the partition on which the message server is

running. If the message server is moved during a relocation event to another host, a license

key for the new environment may be required. Request and import in advance license keys for

all the systems on which the message server may run. The license key depends on the

hardware key: The hardware key is different for different physical servers. (All the LPARs

within one physical IBM Power server have the same hardware key).




The license key is checked during the (re-) start of the message server:

If the relocation is initialized virtualization-based (Live-Partition Mobility), then the

message server keeps running in the same state as before.

o Even if there is no license key for the target server available the SAP system

continuous its operation. The attempt to restart the SAP system on the target

server while having no valid license for the hardware key of the target server

would then fail.

If the relocation is initialized adaptive-based (stop and unprepare on the source host/

prepare and start on the target host), then the message server is restarted on the target

host, and a valid license key for the hardware key of the target server is required.

SAP has introduced a new license key feature called Flexible License Mechanism to simplify

the administration of SAP license keys in system landscapes where the message server can

move between several physical servers. With this method the license key is no longer tied to

the hardware key of the message server. The flexible license mechanism uses a separate ID

generator, which creates a unique network ID for each message server. This network ID is

hardware independent and therefore it is possible to move the message server to a different

host and retain the unique ID. It is possible to configure multiple ID generators to eliminate

any potential single point of failure.

Refer to the current SAP NetWeaver help documentation for a detailed description and

instructions of how to set up this new Flexible License Mechanism:

SAP NetWeaver 7.0 EHP2 and SAP NetWeaver 7.3

3.4.6 Install SAP LVM Prerequisites on the Management Node

3.4.6.1 Storage System Connectivity using SMI-S

As of November 2013 SMI-S connectivity between SAP LVM and storage system is

supported for SAN Volume Controller, IBM Storwize V7000, and IBM System Storage

DS8000. SMI-S connectivity is currently not supported for IBM XIV Storage Systems.

Because of simplicity, SMI-S is the preferred communication method between SAP LVM and

storage systems. SMI-S is already preconfigured on SAN Volume Controller and IBM

Storwize V7000stem and can be used with default users. On IBM System Storage DS8000

one has to configure SMI-S. Please refer to the storage system documentation for more

information.

3.4.6.2 XIV XCLI

The XIV CLI is used for the communication of LPARs with the storage system. It needs to be

available on the management as well as on all managed LPARs.

3.4.6.3 SAN Volume Controller Connectivity

SMI-S is the recommended communication method for SAN Volume Controller.

Nevertheless, in the lab setup we still used the SVC CLI. SAP LVM then communicates with

the IBM SVC storage systems directly using “ssh” connectivity.

The communication is initiated from the central host (the host where the “acsd” component of

IBM Tivoli Storage FlashCopy is installed) only. First a ssh key pair has to be created for the

“root” user on the central system. To allow the execution of CLI commands from SAP LVM

the created key pair must not contain a key passphrase. The private key of the key pair is

http://help.sap.com/saphelp_nw70ehp2/helpdata/en/20/384ef6bd054d5aae1ddd6f67e07a9e/content.htm

http://help.sap.com/saphelp_nw73/helpdata/en/20/384ef6bd054d5aae1ddd6f67e07a9e/content.htm



stored in a common directory. This location is referred as certificate later on in the SAP LVM

storage manager configuration (see Figure 27).

Create a RSA keypair for user root and store the keys e.g. /home/sapadm/.ssh:

> ssh-keygen -t rsa

Generating public/private rsa key pair

Enter file to save the key: /home/sapadm/.ssh/id_rsa

Enter passphrase (empty for no passphrase):

Enter passphrase again:

Your identification has been saved in /home/sapadm/.ssh/id_rsa

Your public key has been saved in /home/sapadm/.ssh/id_rsa.pub

An administrative user needs to be created in the SVC system. This user needs to be member

of the IBM System Storage SAN Volume Controller user group "Administrator".

Transfer the public key of the key pair to the IBM SVC and assign it to the administrative

user.2

Perform a connectivity test after the public key is attached to the SVC admin user:

> ssh -i /home/sapadm/.ssh/id_rsa -l <SVC admin user> <SVC hostname>

2 See the IBM documentation IBM System Storage SAN Volume Controller and IBM Storwize V7000 Version

6.3.0 Command-Line Interface User's Guide (GC27-2287-02): Chapter 1. Preparing the SSH client for details.



4 Initial LVM Configuration

4.1 Engine Settings

The SAP LVM engine settings are normally configured when running the initial configuration

wizard. A number of the settings (for example the default user name or password for the Host

Agents) can be changed again under SetupSettingsEngine.

Figure 19 LVM Engine Settings

4.2 Virtualization Manager Configuration in SAP LVM

The IBM Virtualization Manager settings (IBM Systems Director/ VMControl) are defined on

the InfrastructureVirtualization Managers panel. SAP Note 1728222 – Configuration of

IBM Systems Director VMControl for SAP LVM describes further background information

for this step. The screen shot in Figure 20 shows the start panel. Push the “Add” button to

define a new virtualization manager in SAP LVM.

Figure 20 Virtualization Manager Configuration



Select “IBM Systems Director / IBM Flex System Manager Adapter” and push the “Next”

button.

Figure 21 Virtualization Manager: Virtualization Manager Types

On the next panel enter a label for the virtualization manager, the user name, the password of

the “administration” user that was created in chapter 4.9, the URL to access the virtualization

manager, and the monitoring interval (how often SAP LVM tries to retrieve monitoring data

from virtualization manager)

Press “Test Connection” to validate all the settings and then proceed to the next panel.

Figure 22 Virtualization Manager: Virtualization Manager Properties

Verify the data on the summary screen and press the “Save” button to store the configuration.



Figure 23 Virtualization Manager: Summary

4.3 Storage Manager Configuration

The IBM Storage Manager settings (IBM Tivoli Storage FlashCopy Manager) are defined via

the InfrastructureStorage Managers panel. The start panel is illustrated in Figure 24.

Navigate to this panel, and press “Add” to define a Storage Manager in SAP LVM.

Figure 24 Storage Manager Configuration

In the next screen (Figure 25) the Storage Manager Type can be selected. Select Vendor

“IBM”, Product “IBM Storage Adapter” and click “Next”.

Figure 25 Storage Manager: Storage Manager type



In the next panel (Figure 26) the IBM Tivoli Storage FlashCopy Manager related settings are

specified (See also chapter 3.3.2, as a lot of these parameters need to be provided from the

FCM installation):

The “FCM CLI Host Name” field specifies the node on which the IBM Tivoli Storage

FlashCopy Manager Management agent is installed. The management agent (acsd) controls

the backup flow and mediates between the other agents running on the hosts. The

management agent also provides access to the snapshot backup repository, which contains

information about the valid snapshot backups and their relationships to snapshot capable

storage devices. During a system cloning/copy process SAP LVM starts the management

agent on the “FCM CLI Host Name” node.

The “FCM CLI Path” specifies the working directory for the IBM Tivoli Storage FlashCopy

Manager on all of the nodes. This must be the path that was specified during the installation

customization of the IBM Tivoli Storage FlashCopy Manager (FlashCopy backup for generic

applications in “setup_gen.sh”).

The “IBM FCM template” name is the filename of the profile template located in the “FCM

CLI Path”.

Select “AIX” for the platform type.

The “Storage Connection Method” specifies the storage access method from SAP LVM to the

storage system.

Storage System Storage Connection Method

(SAP LVM) FlashCopy Manager communication to Storage

SAN Volume Controller (V7000), or Storwize V7000

SMI-S (recommended) or Native CLI (SVC CLI via ssh)

SMI-S

IBM XIV Native CLI (XCLI) XCLI

IBM DS8000 SMI-S SMI-S

Table 15 Storage Connection Methods

Figure 26 Storage Manager: Storage Manager Properties

When all required entries are defined, press “Test Configuration”. SAP LVM will initiate a

communication test to FlashCopy Manager and the storage subsystem CLI. The configuration

test should return “Connection successful: SAP LVM successfully gets data from storage

managers through IBM Tivoli Storage FlashCopy Manager”.

After a successful connection test, select “Next” to get to a user and password configuration

screen. Here you can enter the user credentials for the configured storage system. Dependent



on the storage system and the storage connection method you either need to provide a

username and password or the path to a ssh certificate:

Storage System Storage Connection Method

(SAP LVM) User credentials

SAN Volume Controller (V7000), or Storwize V7000

SMI-S Username/ password

Native CLI (SVC CLI) ssh certificate (see 3.4.6.3)

IBM XIV Native CLI (XCLI) Username/ password

IBM DS8000 SMI-S Username/ password

Table 16 Storage Connection user credentials

Figure 27 Storage Manager: Storage Systems

If the user or password is changed in the storage subsystem, you have to update the LVM

Storage Manager configuration too.

Click “Next” and a summary screen appears. Verify the entered settings and click “Save” to

save the configuration.

4.4 Configure SAP LVM Pools

LVM uses pools to separate the landscape components logically and physically. Hosts and

SAP systems are assigned to pools. SAP LVM operations on SAP systems assigned to a

specific pool can only be targeted to hosts that are assigned to the same pool.

Pools are configured on the ConfigurationPools tab. One can define multiple levels of

containers and pools to reflect a hierarchical level of the logical and physical separation of the

landscape entities. The screenshot below shows a container “AIX Pools” groups all pools with

AIX based systems and hosts together. On the next level we separated the pools according to

some physical attributes, all systems and hosts assigned to one pool are allocated or have

physical access to the same storage subsystem.



Figure 28 SAP LVM Pools: Pool Overview

4.5 Configure Networks

SAP LVM requires some network definitions to enable the activation and deactivation of

virtual host names as aliases of existing network interfaces in a host. Navigate to

InfrastructureNetwork ComponentsNetwork and click the “Add” button to define a new

network (

Figure 32).

Figure 29 SAP LVM Network Configuration



Enter a name, the subnet mask, and the broadcast address of the physical network on the

“Basic Configuration” panel.

Figure 30 SAP LVM Network: Basic Configuration

In the next step, “Advanced Configuration” settings are specified. They include the network

gateway address, DNS Zone, Reverse DNS Zone, and Update IP Range.

The “Update IP Range” can restrict the range for new IP addresses that are provisioned by

SAP LVM if automatic DNS update is configured in the environment. If the "Auto Assign IP

Address" option is selected in the "Host Names" step during a clone or copy operation, a

random free IP address is selected from within this range. If an IP address is specified

manually, it must still be within this range.

Figure 31 SAP LVM Network: Advanced Configuration

Click “Next” to proceed to the Summary panel, verify the configuration and push the “Save”

button to store the network configuration.

Figure 32 SAP LVM Network: Summary

4.6 Discovery of Hosts (LPARs)

To discover a new host in SAP LVM, the SAP Host Agent must be installed and running on

the target partition. Select ConfigurationHostsDiscover to get to the “Find



Instances/Hosts” panel. There are various options to perform a host detection, in this example

we select “Detect Using Host and Instance Agent” as Source and “Hosts only” under Find.

Figure 33 SAP LVM hosts: Discover Using Host and Instance Agent

Enter the host name of the partition you want to discover. You can use the default credentials

for the SAP Host Agent user id as configured in the engine settings or choose other

authentication types like specific user/password settings or a X.509 client certificate. Click the

Detect button and you should get a message that new hosts are found.

On the next panel select the pool you want to assign this partition to and click the next button.

Click the Save button on the summary screen to add this newly discovered host.

Figure 34 SAP LVM Hosts: Add Hosts

The next step is to change some properties of this new host. Select the host on the “Hosts

Overview” panel and click “Edit”. The “Basic configuration” step should already have the

correct settings, click Next to get to the “Host properties” panel.



Figure 35 SAP LVM Hosts: Edit host properties

Select the AC-Enabled flag and add a network interface that is used to bind the virtual IP

address of a SAP instance. You also have to provide a capacity figure (SAPS Provided) for

this host. If the partition is used as a target host for cloning and system copy operations, the

“Isolation Ready” flag has to be selected too. Press the “Save” button to store the changes.

4.7 Instance Configuration/ Discovery in SAP LVM

The discovery for a new system is done in several steps. In the “base discovery” the new SAP

system is discovered and added to the SAP LVM configuration. Then the configuration of the

newly discovered instances is completed. This includes the detailed storage configuration for

database and for the central instance.

4.7.1.1 Discovery of a New SAP System

The base discovery of new SAP systems is started on the “Systems” tab in the

“Configuration” view by using the “Discover” button (Figure 36):

Figure 36 SAP LVM Systems: Discover new SAP instances

The new system needs to meet the following prerequisites:

the host is already discovered in SAP LVM

the SAP Host Agent is installed and active on the host

the SAP system is installed in an “adaptive” configuration



o virtual hostnames are defined for both database and central instance

o the storage layout supports separation of database and central instance

the SAP system is active

all the user credentials (e.g. <sid>adm) and their passwords are known to the SAP

LVM admin

For older SAP releases (without having enhanced security measures implemented) the SAP

Host Agent is able to discover both database and SAP instances running on the host. For

recent SAP releases the SAP Host Agent may not be able to detect the SAP instance. Due to

enhanced security settings, the “sapadm” user is not authorized any more to retrieve all the

instance parameters. Discovery of the SAP instance requires to contact the instance agent

using the <sid>adm credentials to retrieve all relevant instance data.

Specify the connection data to the SAP Host Agent (IP address of the host), as well as the

connection details to the instance agent (Figure 37):

Figure 37 SAP LVM Systems: Find Instances/Hosts

The discovery is then started by clicking the “Detect Button”. The database and SAP

instances (central services, application server, diagnostic agent) are identified. Push the

“Next” button to go to the “Add instances” view.

On this screen, the instances are assigned to the appropriate pool.

Verify that the “Host Name” field for each instance matches with the desired virtual name. If

an incorrect host name is suggested, select the action “Modify Host Names”, select the

instance to be changed, enter the correct host name in the “New Host Name” field and click

the “Modify” button to perform the change.



Figure 38 SAP LVM Systems: Add Instances – Modify Host Names

Select action “Auto Assign to New or Existing Systems” once the host names are verified.

Select all instances and choose a target pool using the selection menu in the “Pool for New

Systems”. Click the “Auto Assign” button to assign a System name and the pool to each

instance. Auto assign will generate a new system name for this system

Then click the “Next” button to go to the “Summary” screen and “Save” to save the

configuration.

Figure 39 SAP LVM Systems: Add Instances – Auto Assign to New or Existing System

The base system discovery is now complete. The next steps are to configure additional details

for the SAP system, database, and SAP instances. These additional configuration steps are

required to enable the system for SAP LVM.

4.7.1.2 System Configuration

Select the new system in the “Systems” view and switch to “Edit” mode. On the “Basic

Configuration” screen one can enter a description for the new system, assign a Solution

Manager System and enable E-mail and custom notifications.



Figure 40 SAP LVM Systems: Basic Configuration

Click the “Next” button to go to the “Provisioning & RFC” screen. On this panel, one can

enable the flags for various provisioning scenarios.

If either one of the copying or renaming flags are enabled, a RFC destination (target client(s)

for post copy automation) needs to be defined. The customization panel for a RFC destination

will open automatically by checking the “Configure RFC Destination” flag.

Use the “Test Connection” buttons to verify the data before moving to the next panel with the

“Next” button.

Figure 41 SAP LVM Systems: Provisioning & RFC

ACM was not evaluated in the test installation; use the “Next” button again to go to the next

“Network Isolation” panel.

Network Isolation

In the network isolation step, all network connections which are still allowed in a “Network

fenced” state are configured. The settings can still be changed during the SAP System

Copy/System cloning definition steps. The settings defined here will appear as default

allowed connections during the System Copy / System Cloning step (Figure 42).



Figure 42 SAP LVM Systems: Network Isolation

In the following example, the network connections to “siccserv2” (DNS server for test

environment and the “ssh” protocol were explicitly added as allowed default to the network

fencing configuration.

Figure 43 SAP LVM Systems: Network Isolation - Add allowed communications

Push the “Save” button to save the configuration.

4.7.1.3 DB Configuration

The next step is to configure the detail settings for the SAP database instance. Expand the new

SAP system in the “Systems” tab and select the database.

Press the “Edit” button to open the database configuration panel.



Figure 44 Database Configuration: Instance Details

Basic Configuration

Keep (or change) the settings in the basic configuration view (Figure 45) and then use the

“Next” button to proceed to the “Instance properties” panel.

Figure 45 Database Configuration: Basic configuration

Instance Properties

Set the “AC-Enabled” flag in the Adaptive Enablement section of the screen (Figure 46) and

select the appropriate network. In the “Requirements” section of the screen enter the required

SAPS and Memory figures for the system. These attributes will be used during Start- and

Relocate actions to determine an appropriate target host candidate for the system. The

definition of the target host needs to satisfy the conditions defined in this section. Select the

allowed Operating System levels for the system and proceed with the “Next” button.



Figure 46 Database Configuration: Instance properties

Mount Points

In this step, the storage configuration for the system is defined. This step depends on the

storage provider. For the IBM storage environment, some data needs to be gathered on the

host.

After un-checking the “OS Managed Mounts /Automounter)” flag, the screen changes its

appearance and one can push the “Retrieve Mount List” to gather the currently mounted file

systems from the host.

During a prepare operation the file systems are mounted in the sequence as defined in this

panel. Remove the file systems that are not relevant for the database instance and make sure

that the file systems are defined in the correct order.

Push the “Save” button to store the database configuration.

Figure 47 Database Configuration: Mount points

4.7.1.4 Configuration of SAP Central Services Instance

After the configuration of the database instance is done, the next step will configure details

for the SAP central services instance. Select the “Central services” instance on the “Systems”

tab and push the “Edit” button. Then follow the same steps as in the database configuration

section to configure the attributes of the central services instance.

This time make sure to choose the appropriate file systems required for the central service

instance on the “Mount points” panel.



Figure 48 Central Services: Mount points

4.7.1.5 Configuration of SAP Primary Application Server Instance

Follow the same steps once more to configure the primary application server instance for this

system (SAP LVM still uses the term “Central Instance” to designate the primary application

server instance).

At this time all configuration steps for a new system are complete and the system should show

up as operational under OperationsSystems tab.

Figure 49 Overview of systems and instances

4.8 Additional Configuration Steps

4.8.1 Configuration for SAP System Copy

During a SAP System Copy and a SAP System Rename activity SAP LVM starts the SAPInst

rename tool on the target LPAR to change the SAP System ID of the system. SAP LVM

expects that the SAPInst tool is available in an installation folder accessible on the target

LPAR. In the test environment a central NFS server is used as repository, and the directories

are mounted on all LPARs. This information is defined in ConfigurationExtended

SettingsSystem and Provisioning for each different target OS type. Click the “Add” button

to add a new “System Copy” definition.



Figure 50 System and AS Provisioning Configuration

Select “System Copy” as provisioning type and proceed with the “Next” button.

Figure 51 System Copy Configuration: Release configuration type

On the next screen enter a descriptive name and select the release version and operating

system type.

Figure 52 System Copy Configuration: Basic properties

Enter the path to the System Rename tool and proceed to the Summary panel. Verify the

configuration on the Summary screen and click “Save” to store the configuration.

Figure 53 System Copy Configuration: Installation master

4.9 IBM Flex System Manager Configuration

IBM Flex System Manager is delivered as a pre-installed appliance node for an IBM PureFlex

System. FSM integrates the virtualization management functionality of VMControl, which

simplifies the management of virtual resources (server, storage, network, virtual appliance

images) and pools of virtual resources.



The following shows the basic steps required to enable VMControl for virtual appliance

image management:

Discover storage and configure storage system pools

Discover servers and configure server system pools

Configure fabric management for SAN switches (a deploy operation might have to

update the SAN zoning)

Discover, request access, and collect full inventories on each object used by

VMcontrol (server pools, storage pools, physical server, VIO servers, NIM server, etc)

Install the IBM System Director Common Agent, the NIM subagent, and common

repository subagent on the NIM server

Create a common repository on the NIM server

Capture or import a virtual appliance

Please refer to the redbook “IBM Systems Director VMControl Implementation Guide on

IBM Power Systems” (http://www.redbooks.ibm.com/abstracts/sg247829.html?Open) and the

VMControl documentation for a detailed description about implementing VMControl and its

image deployment capabilities.

Once VMControl is configured and the Flex System Manager is defined in SAP LVM, a

subset of the available tasks (like capture or deploy an image) can be started directly from the

SAP LVM user interface too.

4.9.1 Setup of IBM System Management Security

It is possible to use IBM Flex System Manager or IBM Systems Director administrator users

for integration with SAP LVM. Some virtualization use cases (OS provisioning and LPAR

relocation using Live Partition Mobility) require administration credentials. For virtualization

monitoring and management, it is also possible to use credentials that are more restricted.

Administrators may allow monitoring and management of particular elements only (e.g. of

certain physical servers, LPARs, operating systems, etc.) and thus restrict the view of SAP

LVM users.

Please check sap note SAP note 1728222 for the latest information about the definition of

virtualization manager users and roles.

Below we describe how to extend IBM Flex System Manager roles and groups for integration

with SAP LVM.

1. Logon to the FSM console and navigate to AdministrationView or modify user

accounts. Switch to the Groups tab and use the Create button to create a new group.

2. Go back to the User tab and create a new user. On the “User groups” panel select the

previously created group.


http://pic.dhe.ibm.com/infocenter/director/pubs/index.jsp?topic=%2Fcom.ibm.director.vim.helps.doc%2Ffsd0_vim_main.html

https://websmp109.sap-ag.de/~form/handler?_APP=01100107900000000342&_EVENT=REDIR&_NNUM=1728222&_NLANG=E



Figure 54 IBM Flex System Manager Security: Group membership

3. Go to AdministrationManage Roles, select the SMUser role and use the “Create Like”

button to create a copy of the role, for example create a new copy called LVM_User.

Perform the same step to create a copy of the SMManager role. We named this copy

LVM_Manager.

Figure 55 IBM Flex System Manager Security: Create new roles

4. Select the new LVM_User role, click “Edit” and extend the copied role with following

permissions (select the permission under “Available permissions” and click “Add >” to

move it to “Selected permissions”):

GeneralCreate Group

GeneralEdit… (Description: Provides an editor for changing an existing group)

System Status and HealthMonitors



Figure 56 IBM Flex System Manager Security: Modify role permissions

5. Use the same steps again to extend the copied LVM_Manager role with following

permissions:

Power On/OffPower Off Now

Power On/OffPower On

Power On/OffShut Down

Power On/OffShut down and power off

6. Navigate to Plug-insDiscovery ManagerResource Explorer and create two static

resource groups for monitoring and management.

The monitoring group should include physical machines, virtual appliances, storage pools,

virtual servers, and all other resources that should be visible in SAP LVM.

The management group should include only the virtual servers that will be manageable

through SAP LVM.



Figure 57 IBM Flex System Manager Security: Define static groups

7. Assign the new roles and resource groups directly to the user or the user group created in

step 1 and 2. In this example, we use the user group.

Navigate back to AdministrationView or modify user accounts. Switch to the “Groups”

tab, select the SAP LVM group and click “Assign Role”.

Select the LVM_User role under “User Role” and the “LVM_Monitoring” group under

“Selected resource groups”. Click the “Add >” button to add this combination to the

“Assigned Roles”.

Perform the same step with the copy of the SMManagement role and the

“LVM_Management group”.

Figure 58 IBM Flex System Manager Security: Assign groups and roles to user group

The configured user is now available for the integrating IBM Flex System Manager as a

virtualization manager in SAP LVM.



5 Detailed Description / Execution of Use Cases

5.1 LVM Standard Edition Use Cases

This section describes the use cases included in the standard version of SAP LVM. Basically

the SAP LVM Standard Edition allows to start, stop, and relocate SAP Systems.

5.1.1 Description of the LVM Standard Edition Use Cases

Following use cases are possible:

Prepare/unprepare SAP instances and systems

Start/stop SAP instances and systems

Relocate SAP instances and systems

Start/stop hosts

Monitoring

5.1.1.1 Prepare/Unprepare SAP Instances and Systems

The Standard Edition of SAP LVM is able to prepare and unprepare SAP systems and

instances. In case of a prepare operation, the SAP instance or system is made ready to be

started on a host. During the prepare phase, the storage volumes belonging to an SAP system

are mapped to the LPAR, the volume groups are imported, the file systems are mounted, the

network isolation on the LPAR is being setup, and the virtual hostnames are assigned to a

network interface.

During an unprepared operation, file systems volumes belonging to a SAP system are

unmounted, the volume groups are exported, the storage volumes are unmapped, and the

network isolation and virtual hostnames are being removed.

Prerequisite is that the SAP instance or system to be unprepared is stopped. This allows to

map and start the SAP system to/on the same host, or to/on another host.

The prepare/unprepare can be run on a SAP instance or on a complete SAP system including

all instances belonging to an SAP system. If the complete SAP system is affected, then the

function is called mass operation.

Prepare/unprepare triggers following calls:

Storage CLI commands in order to attach or detach storage volumes containing SAP

systems to / from certain OS hosts

OS specific storage libraries for handling the storage mapping

OS specific libraries for handling the SAP instance service

OS specific libraries for handling the network setup

5.1.1.2 Start/Stop SAP Systems and Instances

In order to start a SAP instance or system, it needs to be successfully prepared. After a SAP

instance or system has been stopped, it can be unprepared.

The prepare/start and stop/unprepare tasks can be executed as a single workflows.

As for prepare/unprepare mass operations allow to start/stop complete SAP systems versus

single SAP instances.

Start / stop triggers following calls:

OS specific libraries for handling the SAP tasks start / stop

5.1.1.3 Relocate SAP Instances and Systems

The relocation of SAP instances and systems allows to move SAP instances or systems to

another host.

There are two flavors of SAP instance or system relocation:



IBM PowerVM based relocation

The complete LPAR including the SAP system and operating system is moved (also

called migrated) from one physical host to another physical host. In that case, LVM

uses IBM PowerVM Live Partition Mobility (LPM). The LPAR and SAP system stay

up and running during the execution of LPM. LPM is used e.g. for hardware

maintenance activities.

Hypervisor based relocation triggers following calls:

o IBM PowerVM specific libraries for handling Live Partition Mobility in the

HMC or Systems Director

SAP LVM relocation

This means that the SAP system is moved from one managed host to another managed

host. This causes a short SAP downtime as the SAP system needs to be stopped on the

source host and started on the target host. The process automates following functions:

o Stop SAP instance or system

o Unprepare SAP instance or system

o Prepare SAP instance or system on the ‘new’ LPAR

o Start SAP instance or system on the ‘new’ LPAR

As for prepare/unprepare or start/stop, mass operations allow to relocate complete

SAP systems versus single SAP instances.

SAP based relocation can be used for operating system maintenance.

SAP based relocation triggers following calls:

o Storage CLI commands in order to attach or detach storage volumes containing

SAP systems to/from certain OS hosts

o OS specific storage libraries for handling the storage mapping

o OS specific libraries for handling the SAP instance service

Figure 59 and Figure 60 illustrate the two relocation scenarios.



Figure 59 IBM PowerVM Relocate

Figure 60 SAP LVM Relocate



5.1.1.4 Start/Stop Hosts

With SAP LVM hosts can be started or stopped.

Caution: Powering off hosts without stopping and unpreparing the SAP systems running

inside the LPAR may cause data loss or data corruption.

Starting or stopping LPARs triggers following calls:

IBM PowerVM specific libraries for handling Live Partition Mobility in the HMC or

Systems Director

5.1.1.5 Monitoring

SAP LVM provides several types and levels of monitoring, e.g. CPU consumption, memory

consumption, dashboard, history data etc.

SAP LVM communicates with the virtualization manager to retrieve monitoring data about

IBM Power physical servers and logical partitions. Optionally, SAP LVM tries to gather CPU

utilization data for discovered elements (physical servers and LPARs). SAP Notes 1728222

(for IBM Systems Director VMControl adapter) and 1728293 (for IBM HMC adapter)

provide configuration details of components involved in this scenario.

Following setups are supported:

Monitoring through IBM Systems Director VMControl adapter

Monitoring through HMC adapter

5.1.2 Executing the LVM Standard Edition Use Cases

5.1.2.1 Prepare/Unprepare SAP Instances and Systems

The following section describes the required steps to prepare a SAP instance. As the

unprepare process is very similar, it will not be shown separately.

As a prerequisite, the SAP system needs to be installed and discovered in SAP LVM with

correct storage volumes and mount points.

To prepare a SAP instance:

Log in to SAP LVM and navigate to OperationsSystems. Select the pool in which the SAP

system resides, then the SAP system you want to prepare. Select the instance and assign a

host in the Host Properties field. Click the small arrow in the right lower corner of the button

‘Prepare and Stop’ button and select ‘Prepare‘.



Figure 61 Prepare SAP Instance

You can monitor the progress in the ‘Logs’ tab (see Figure 63).

5.1.2.2 Start/Stop SAP Systems and Instances

The following section describes the required steps to start a complete SAP system including

the prepare step. As the complete SAP system is started, this is a mass operation.

The stopping process is very similar, it will not be shown separately.


correct storage volumes and mount points.

To start a complete SAP instance including preparation:


system resides, then the SAP system you want to start. Select or enter a host for each instance

to be started and select the operation ‘Mass Start (including prepare if possible)’. Click the

‘Execute’ button to start the operation.



Figure 62 Start SAP system: Mass Operations

The logs for an operation are available under MonitoringLogs:

Figure 63 Logs of a 'Start SAP system' Operation



5.1.2.3 Relocate SAP Instances and Systems

The following section describes the required steps to relocate a complete SAP system using

SAP LVM relocation. That is, only the SAP system (not the host) is moved to another LPAR.

As the complete SAP system is relocated, this is a mass operation.


correct storage volumes and mount points, and it needs to be running.

To start the relocation of a complete SAP system:


system resides, then the SAP system you want to relocate. Select the instances and select a

target host for each instance. Select the operation Select “Mass Relocate All in State

‘Running’’’ and click the ‘Execute’ button.

Figure 64 Relocate SAP System: Mass Operation



5.1.2.4 Start/Stop Hosts

The following section describes the required steps to stop a LPAR. As the start process is very

similar, it will not be shown separately.

Make sure that all SAP systems and instances are stopped and unprepared. Otherwise, you

risk a data loss or a database corruption. There are two options to stop a LPAR:

1. OS Shutdown: LVM triggers an operating system shutdown

2. Power Off: LVM just switches off the LPAR

Using the OS Shutdown is recommended. Use Power Off only in case the shutdown is no

longer working, e.g. the LPAR is not reacting to any commands. Power Off will not close

network connections, unmounts file system etc.

To shut down a host:

Log in to SAP LVM and navigate to OperationsHosts. Select the pool in which the host

resides, then the host you want to stop. Select the ‘Virtual Host’ tab, then ‘Deactivate’ and

select Deactivate (OS Shutdown).

Figure 65 Shutdown Host



5.1.2.5 Monitoring

The following section describes how to access the different monitoring functions.

Dashboard

The Dashboard is the first screen opened per default when you log in to LVM. It provides

information e.g. regarding SAP systems, physical resources, and virtual resources. By

selecting ‘Add Pool’ you can open a ‘window’ showing an additional subset of information.

Figure 66 SAP LVM Dashboard

Visualization

The ‘Visualization’ tab provides a visual view on the infrastructure and SAP systems and

instances running on the infrastructure.



Figure 67 SAP LVM Infrastructure Visualization

Performance monitoring

Performance monitoring enables you to display metrics for specific entities (such as SAP

systems, specific instances, or virtual machines) for a specific time period. The historical

monitoring data is fed to automatic capacity management (ACM) to provide response time

data.

To open the performance monitor:

Log in to SAP LVM and navigate to MonitoringPerformance.

Select the pool in which the host resides, then the host you want to stop. Select the ‘Virtual

Host’ tab, then ‘Deactivate’ and select Deactivate (OS Shutdown). Select one or more entities

on the left (for example, application server) and click the metric you want to display (e.g CPU

usage) and the period (e.g. hour).

To add entities to the display of a metric, specify the entity in the Entity field (or choose the

entity using the value selector) and choose Add.



To add the metric to the dashboard as a pod, click the ‘Add to Dashboard’ button. If you have

added entities to the display of the metric, the metric is displayed on the dashboard only for

the first entity.

Figure 68 SAP LVM Performance Monitoring

5.2 SAP LVM Enterprise Edition Use Cases

The execution of SAP LVM Enterprise edition use cases require that

LVM Enterprise Edition (VCM_ENT.SCA) is deployed to the managing SAP

NetWeaver Java Web AS system.

5.2.1 SAP System Copy Use Cases

The SAP LVM Enterprise Edition Use-Cases include the SAP System Copy use cases

Figure 69 illustrates the three different scenarios for SAP System copies in SAP LVM:

SAP System Clone

Initial SAP System Copy

SAP System Copy Refresh



Figure 69 SAP System Copy Scenarios Supported by SAP LVM (Source SAP)

5.2.1.1 SAP System Clone

In the SAP System Clone scenario, an exact duplicate of an existing SAP system is created.

Content and system configuration are 100% identical in all aspects to the original system. To

avoid conflicts, e.g. with interfaces dealing with the original system landscape, the clone SAP

system is fully network isolated from the rest of the environment before the SAP target

system can be started. Therefore, the clone system is established in an isolated networking

environment. AIX IPsec allows restricting the outbound TCP/IP network communication to

hosts/ports which don’t participate in any interface communication of the SAP system.



SAP System cloning is realized in two different versions:

Storage based System Clone

For storage based cloning the system to be cloned must be “adaptive enabled”: This

includes

o virtual IP addresses for both database and central instance

o Storage Setup separating all the SAP and database entities from the operating

system image. The SAP and database entities must be deployed to storage

supported by the SAP LVM storage adapter.

The target host must match all prerequisites for a relocation target of adaptively

installed systems (e.g. users, services, etc).

Virtualization based

For virtualization based System Cloning the system to be cloned must be installed in

the rootvg of a LPAR that is managed by the SAP LVM virtualization manager, and

which is registered with SAP LVM. In addition, virtualization based cloning is

supported with an “offline database copy” only. During virtualization based System

Cloning, the SAP System is stopped, the LPAR is captured via IBM Systems Director/

VMControl, and is deployed to the new target LPAR afterwards.

5.2.1.2 Initial SAP System Copy

During a SAP System Copy a duplicate of an existing SAP system is created with a different

system name and a unique SAP system ID (SID). The target system is updated with the new

system information, and can run in the same network environment as original/source SAP

system.

As the first step, the system copy involves the creation of a system clone to create the initial

target system. Then this initial target system is modified during subsequent steps to update the

settings according to the new SAP System ID. A special version of SAPInst, “SAPInst

Rename” is used to change the SAP system ID during this process. Additional post

configuration tasks in the SAP System are achieved via the Post Copy Automation (PCA)

tool. PCA provides task lists, with a predefined sequence of configuration tasks to configure

the extensive technical scenarios automatically. In the SAP LVM environment the execution

of a PCA task list is embedded in the overall system copy workflow.

5.2.1.3 SAP System Refresh

Given are two systems 'SAP source' and 'SAP target' which exist and are configured in a

copy-relationship (target is copied from source). SAP LVM then allows to 'refresh' the 'target'

system by:

exporting configuration data from the 'target' system

SAP System Copy for copying the source system to the 'target' system

(same target SID, same hostnames as previous system copy)

importing the formerly exported configuration data into the new 'target' system

The System Refresh feature has the same requirements as the System Copy.

5.2.1.4 SAP System Rename

The SAP System Copy can be executed in two steps also:

Create a SAP System Clone in a first step

Then use this clone later on, and rename it to a new SAP System ID. Like in scenario

5.2.1.2, the SAPInst Rename tool is invoked for the SAP System ID change.



At first, the SAP system to be renamed needs to be enabled for the SAP System Rename by

executing following steps:

Go to ConfigurationSystem. Select the new cloned System and click the “Edit” button.

In the configuration screen, activate the checkbox “Renaming” and click “Save”.

Figure 70 Enable SAP System for SAP System Rename

5.2.2 SAP ACM (Automatic Capacity Management) Use Cases

The SAP ACM use cases are SAP LVM enterprise use cases too. However, they will not be

discussed in further detail here. Just note that the following considerations need to be taken

into account for the ACM Use-Cases:

Both adaptively installed SAP systems and traditionally installed SAP systems are

supported for ACM

One dialog instance per host is supported

All ACM managed dialog instances must be in the same set of logon groups reserved

for use by ACM.

SAP CCMS must be installed and operational for each managed SAP system.

For ACM-managed VMs in the same LVM pool, all the OS users must be configured

identical and must be defined in the central user store (LDAP).



5.2.3 Required Preparation Steps

Table 17 includes an overview about the required initial setup step before a system can be

cloned, copied, refreshed or renamed with SAP LVM. For each of the required steps it

references to the section in this document where the step is described in more detail.

Setup Procedure in SAP LVM

Clone System Copy System Refresh System/ Database

Rename System

Configuring Networks

Required (See chapter 4.5)

Configuring Storage Providers

Required, if the source system is adaptively installed. (See chapter 4.3)

Not Required

Configuring User Management

Not part of the test environment: All users for the target systems had been pre-defined in LDAP

Configuring Name Servers

Not part of the test environment: All virtual hostnames and IP addresses had been pre-defined in DNS.

(See chapter 5.2.4.1 for additional information)

Configuring Remote Function Call Destinations

Not Required Required

(See chapter 5.2.5.2)

Setting the Infrastructure Assignment

Required (See chapter 4.7.1.1 and 4.7.1.2)

Configuring System Rename Configurations

Not Required Required

(See chapter 4.8.1)

Configuring Mount Points

Required, if the source system is adaptively installed (See chapter 4.7.1.3 and 4.7.1.4)

Configuring Java Post Copy Automation

Not Required Not part of the test environment:

SAP ABAP Stack tests only

Configuring a universally unique identifier (UUID)

Required (See chapter 3.4.5.4)

Not Required

Enabling the system for (See chapter 5.2.1.4, Figure 70)

cloning copying refreshing renaming

Table 17 Required Setup Procedures

5.2.4 Execution of a SAP System Clone Operation

The following section describes the required steps for starting the SAP system clone in SAP

LVM. As a prerequisite, a SAP source system need to be installed and discovered in SAP

LVM, and need to be enabled for SAP System cloning in the SAP LVM configuration

A SAP System Clone is started on the “System and AS Provisioning” tab in the

“Provisioning” view. Select the SAP System to be cloned and click the “Clone System”

button.



Figure 71 Start SAP System Clone

SAP LVM starts the System Cloning dialog. In the first screen, the LVM Pool is shown, and a

short name and a description for the target system need to be specified. Accept the default

settings, and click “Next” to go to the next screen.

Figure 72 SAP System Clone: Basic Data of Target System

In the next screen, target hosts for the SAP system clone are specified:

1. In this example the default value “Use Existing Host” was chosen for the Central

Services instance and the Central instance. SAP LVM takes an existing LPAR out of

the pool of hosts. The instances will be cloned and attached to this already existing

host.

For the Database instance, “Provision of New Host” is selected. Here a new LPAR

will be created as the target host for the SAP System Clone. Once you click “Add”,

SAP LVM calls the virtualization manager (in our case IBM Flex System Manager

VMControl) to deploy the new target host as part of the cloning process.

2. Click “Next” to continue with the next selection screen



Figure 73 SAP System Clone: Host Selection of Target System

In the next screen, the virtual host names and the network for the SAP system clone are

defined. The virtual hostname per default is derived from naming conventions. The short

name specified in Figure 72 is inserted into the virtual hostname.

5.2.4.1 Excursus: DNS Name Resolution

DNS Name Server Considerations

In our test landscape all target IP addresses for the “clone” IP labels are predefined in DNS by

an appropriate address range.

Role IP Label (DNS server) IP Address

SAP Source cl6ci01 192.168.165.116

cl6db01 192.168.165.126

SAP Target cl6cloneci01 192.168.165.136

cl6clonedb01 192.168.165.146

Table 18 Examples for IP addresses

SAP Note 1572841 - Setup of name server update of SAP LVM describes an alternative

solution where SAP LVM automatically updates/creates virtual hostnames in the DNS server.

SAP LVM then creates the respective A (forward) and PTR (reverse) lookup records for IPv4

addresses in the DNS server at the runtime during the SAP System Clone operation. During a

SAP System Destroy operation these records previously added are removed. This mechanism

ensures that in the DNS server the new IP addresses are assigned to the new IP label. In the

test environment however all virtual hostnames and IP addresses had been pre-defined in

DNS and the automatic name server update was not tested.

DNS Behaviour on the Target Host

On the target host, things may look slightly different. SAP LVM acquires the new IP

addresses on the target host during the “prepare” operation, e.g.

On the target host:

root> netstat -in

Name Mtu Network Address

en0 1500 link#2 86.1f.b8.54.6b.2

en0 1500 9.153.164 9.153.164.137

en0 1500 192.168.164 192.168.165.146

en0 1500 192.168.164 192.168.165.136




However, the name resolution on the target host still states the “source” IP labels. This is

required for SAP system operation of the clone. The clone system is a 100% identical clone of

the source system and the virtual IP labels need to be unchanged on the target host (as e.g. the

virtual IP labels appear in the instance profiles, start scripts, etc.).

On the target host:

root> netstat -in

Name Mtu Network Address

en0 1500 link#2 86.1f.b8.54.6b.2

en0 1500 9.153.164 siccp137

en0 1500 192.168.164 cl6db01

en0 1500 192.168.164 cl6ci01

To achieve this behaviour, SAP LVM adds new entries to the local /etc/hosts file:

On the target host:

root> cat /etc/hosts

…

192.168.165.136 cl6ci01.isicc.de.ibm.com cl6ci01 #SAP-LM: Added to

modify local hostname resolution for cloning DO NOT MODIFY

192.168.165.146 cl6db01.isicc.de.ibm.com cl6db01 #SAP-LM: Added to

modify local hostname resolution for cloning DO NOT MODIFY

With “local” name resolution, the new “target” IP addresses are mapped to the “source” IP

labels. In order to ensure that the mechanism works, it is mandatory that the DNS search order

evaluates the local /etc/hosts first. This ensures that the local host file supersedes the

(external) nameserver. The nameserver gets contacted only in case the record to be resolved is

not found in the local hosts file.

root> cat /etc/netsvc.conf

# order of search: 1. /etc/hosts 2. nameserver

hosts = local,bind4

Back to the selection screen of the SAP system clones, see Figure 74.

1. The network name needs to be selected

2. Click “Next” to continue with the next screen



Figure 74 SAP System Clone: Virtual Host Names and Networks

In the next screen, all the storage definitions are made for the SAP system clone.

1. SAP LVM provides proposals based on built in naming conventions for the new

volume names on the storage system, and logical volume names on the operating

system. You can accept the defaults, or overwrite the values based on your own

naming conventions.

2. Selection of “Full Clone” allows to create a new, independent set of target volumes

(which no longer depend on the source volume)

3. SAP LVM also allows specifying the storage pool on the storage systems which

should be used the hold the target system. If you don’t specify it, then LVM will use

the same storage pool where the source system is located.

4. Click “Next” to get to the next selection screen.

Figure 75 SAP System Clone: Storage Volumes



On the next screen, specify the database consistency method.

By selecting “Online Copy/Suspend IO Mode”, SAP LVM will place the DB2 UDB database

into “write suspend” state before the FlashCopy/SnapShot is triggered in the storage system.

When the FlashCopy/SnapShot is finished, SAP LVM will initiate a “write resume” to bring

back the database into the operational state.

Figure 76 SAP System Clone: Database consistency

In the next screen, all the port definitions for the network fencing of the SAP system clone are

specified.

The highlighted item under allowed connections is the DNS name server. In our test

environment, the standard /etc/hosts file didn’t include entries for the virtual IP labels. All the

name resolution is done via DNS to the DNS server. So it is important that the LPAR is able

to communicate with the DNS server in a fenced state too. Therefore the DNS server (here:

siccserv2) was added as allowed communication already during the SAP system setup in SAP

LVM.

Click the “Next” button to get to the final confirmation screen.

Figure 77 SAP System Clone: Define Allowed Outgoing Connections for System Isolation



The final confirmation screen appears. It is possible to review all the parameters defined in

the steps before.

Click “Start System Cloning” to finally start SAP System Clone execution.

Figure 78 SAP System Clone: Summary

After starting the SAP System Cloning process, the Monitoring screen is launched and the

progress of the SAP System Clone can be tracked.

5.2.5 Execution of a SAP System Copy

The following section describes the required steps for starting the SAP system copy in SAP

LVM. As a prerequisite, a SAP source system needs to be available and discovered in SAP

LVM, and need to be enabled for SAP system copies in the SAP LVM configuration.

5.2.5.1 Preparations for Post Copy Automation (PCA)

Post Copy Automation (PCA) is part of SAP NetWeaver Landscape Virtualization

Management, enterprise edition solution. This section illustrates how to set-up PCA initially.

Several preparation steps before set-up of SAP Post Copy Automation need to be executed in

the source system first: Details are described in the document "ABAP Post Copy Automation

Guide for Enterprise Edition". See SAP NetWeaver Landscape Virtualization Management -

Installation & Upgrade GuidesSAP NetWeaver Landscape Virtualization Management 2.0

and in SAP Note 1614266 - System Copy: Post Copy Automation (PCA) / LVM for further

details.






The file "Note_Overview.pdf" (which is attached to SAP Note 1614266) lists a set of

additional SAP notes which need to be implemented as a prerequisite for PCA in the source

system.

Dependent on the actual support package stack of the SAP system they may be already

included in the system, and corrections may need to be applied to the SAP system (via

transaction SNOTES). The ABAP program Z_SAP_NOTE_ANALYZER (which is also

attached to SAP Note 1614266) automates this procedure. The SAP system is checked, and

according to its actual support package stack the required SAP notes content is downloaded

directly during execution of the from Z_SAP_NOTE_ANALYZER report, the content is

transferred into the SNOTES notes assistant transaction and added to the queue.

If all the pre-requisites are met, then PCA is installed via the “Post Copy Automation

Installer” (PCAI) automatically into the managed source system.

See also the following notes and the referenced notes that are mentioned in there:

Note 1589145 - Task Manager for Technical Configuration,

Note 1589175 - System Copy: Task Content for Task Manager

Further details are described in chapter 3 and chapter 4 of the “ABAP Post Copy Automation

Guide for Enterprise Edition“.

PCA is part of the enterprise edition of the SAP NetWeaver Landscape Virtualization

Management solution. PCA is shipped with SAP NetWeaver packages, but is deactivated per

default. The License Enabler add-on PCAI_ENT 1.0 activates PCA and is required in the

system to enable the execution of PCA task lists. Install the add-on PCAI_ENT 1.0 before the

first system copy of the system using transaction SAINT. Details are described in chapter 5 of

the “ABAP Post Copy Automation Guide for Enterprise Edition“.

To grant the privileges to execute PCA activities for the administrator (dialog user) the

corresponding roles SAP_BC_STC need to be assigned to all relevant users. This includes the

system administrator user responsible for PCA activities, and the technical users that will be

used via remote access by SAP LVM. Details are described in chapter 7 of the “ABAP Post

Copy Automation Guide for Enterprise Edition“.

PCA provides task lists, with a predefined sequence of configuration tasks to configure

extensive technical scenarios automatically. To prepare the run of these task lists, the ABAP-

based “task manager for technical configuration” is used. In the SAP LVM environment, the

execution of such a task list is induced without calling the task manager for technical

configuration directly. Further details are described in chapter 8 and chapter 9 of the “ABAP

Post Copy Automation Guide for Enterprise Edition“.

5.2.5.2 Add the RFC destinations for PCA to the SAP System configuration

In the systems configuration, define the RFC destinations that will be used for the PCA

execution for the SAP system (see Figure 79).

Define the logon parameters for the main destinations, and click “Test Connection”. An

attempt to connect to the SAP system is made.

Press “Retrieve Clients” and the list will be populated with all clients of the existing system.

Different PCA activities may be executed specific for each client of the SAP system.

Adjust the Users accordingly, and define the passwords.





Figure 79 Define RFC Destination for PCA Execution

5.2.5.3 Start the SAP System Copy

A SAP System Copy is started in the “System and AS Provisioning” tab in the “Provisioning”

view. Select the SAP system to be copied and click the “Copy System” button.

Figure 80 Start SAP System Copy



On the next screen, enter the SAP system ID and master password for the target system.

Both will be used later during the “SAPINST_RENAME” step to adapt the configuration. The

additional description may be adjusted additionally. This will be the description for the new

system in the SAP LVM configuration.

Click the “Next” button to go to the next step.

Figure 81 SAP System Copy: Basic Data of Target System

In the second step the target host(s) for the system copy is defined:

1. In this example the default value “Use Existing Host” was chosen. SAP LVM takes an

existing host out of the pool of hosts. The instances will be cloned and attached to this

already existing host.

2. Alternatively, you may select “Provision of New Host” for one or more instances. In

this case, a new LPAR would be created as the target host for the SAP instance(s).

SAP LVM would call the virtualization manager, here IBM Flex System Manager

VMControl, to deploy the new target host as part of the SAP system copy process.

3. Click the “Next” button to continue with the next selection screen

Figure 82 SAP System Copy: Host Selection of Target System

In the next step, the network configuration for the target system is specified.



The IP addresses for the target system are proposed according to the naming conventions for

the virtual IP labels (<sid>pasXX, (<sid>scsXX, <sid>dbsXX). The addresses and labels are

either predefined in the DNS (respectively in the local host files of the SAP LVM system and

the involved resources) already.

Otherwise, SAP LVM may directly update the DNS domain name server for the new entries,

however that need to be defined and configured according to SAP Note 1572841 - Setup of

name server update of SAP LVM first.

A network name for the Network needs to be selected by using the drop-down list.

Click “Next” to continue with the next step.

Figure 83 SAP System Copy: Virtual Host Names and Networks

In Step 4, the Instance number(s) for the Instances of the target system are defined. Per

default, the target system will be installed using the same instance numbers as the source

system. Click “Next” to continue.

Figure 84 SAP System Copy: SAP Instance Numbers

In step 5 the storage configuration for the target system is defined. The storage image for the

target system is created as a SnapShot/FlashCopy of all source volumes to the target volumes.

The main structure like the overall number of AIX volume groups and their content cannot be

changed. Names for volume groups, logical volumes and filesystem names can be adjusted

however. The storage pool for the target volumes can be specified.





Per default, all SAP System IDs in the filesystem names are updated to the new SAP System

ID of the target system.

Figure 85 SAP System Copy: Storage Volumes

In case the storage connection to the IBM SVC is configured to use ssh, a warning message is

shown, as the recommendation is to use SMI-S.

Click “Ignore warnings for the step” and continue with “Next” button.

Figure 86 SAP System Copy: SVC CLI Warning Message

In the next screen, specify the database consistency method.




back the database into operational state.



Figure 87 SAP System Copy: Database consistency

In step 7 the OS administration users are specified for the target SAP system. If LDAP is used

for central user management, then the users need to be created manually before the SAP

system Copy is started. “sapinst” is currently not able to handle LDAP users correctly.

However, SAP LVM is able to create local users as part of the SAPInst Rename step.

Click “Next” to continue the next step.

Figure 88 SAP System Copy: User and Group Management

In step 8 further additional settings for DB2 schema user and DB2 instance are specified for

the new SAP System ID. For DB2 UDB, the schema user cannot be changed during the

process and must be set identical to the schema user of the source database. The name of the

database instance for the target system is derived from the SAP System ID according to SAP

naming conventions.

The DB2 Security Administrator can be replaced (from db2<src-sid> to db2<tgt-sid>), which

will cause a re-assignment of all DB2 objects to the new security admin.



Click “Next” to continue.

Figure 89 SAP System Copy: SAPinst Release Configuration

In step 9, the allowed ports/connections for the network fencing step are defined. Additional

ports/hosts allowed for communication can be selected out of the set of “Current connections

on host” from the source system, and then added to the “allowed outgoing connections on the

target hosts”.

To avoid additional manual steps it is best-practise to include all the necessary hosts/ports

required for outbound connections of the system already in the SAP system definition.

Click “Next” to go to the next step.

Figure 90 SAP System Copy: Define Allowed Outgoing Connections for System Isolation

In step 10, the SAP clients for executing the Post-Copy Automation (PCA) steps can be

defined.



Figure 91 SAP System Copy: Post Copy Automation

All available tasks can be gathered from the source system, and can be selected accordingly.

Click the “Next” button to continue to the next selection screen.

Figure 92 SAP System Copy: Select Task List

Before actually starting the SAP System Copy, a summary screen is displayed. All the

parameter settings can be reviewed by expanding the different subsections, and eventually

adjust them once again.



Figure 93 SAP System Copy: Summary View

Click “Start System Copy” to finally start the SAP System Copy. The Monitoring screen is

launched and the progress of the system copy can be tracked.

5.2.6 Execution of SAP System Refresh

Prerequisite for the SAP System Refresh is a SAP System Copy run that was previously

completed successfully.

The execution of the SAP System Rename is very similar to the steps before. Start with the

Provisioning view. Click the “Refresh System” button to start the parameter dialog.



Figure 94 SAP System Refresh: Start System Refresh

In the first screen base configuration parameters can be adjusted.

1. Specify and confirm the Master Password

2. Click “Next” to continue with the next selection screen

Figure 95 SAP System Refresh: Basic Data of System to be Refreshed

Acknowledge the target host selection in the following screen and click “Next” again.



Figure 96 SAP System Refresh: Host Selection of Target System

Acknowledge the virtual hostnames and network settings and click “Next”.

Figure 97 SAP System Refresh: Virtual Host Names and Networks

In the next screen, all the storage definitions are made for the SAP system clone:

1. SAP LVM provides proposals based on built in naming conventions for the new

Volume names on the storage system, and Logical Volume names on the operating

system. You can accept the defaults, or overwrite the values based on your own

naming conventions.

2. Selection of “Full Clone” allows to create a new, independent set of target volumes

(which no longer depend on the source volume)

3. SAP LVM also allows specifying the storage pool on the storage systems which

should be used the hold the target system. If you don’t specify it, then LVM will use

the same storage pool where the source system is located.

4. Click “Next” to get to the next selection screen.



Figure 98 SAP System Refresh: Storage Volumes

In the next screen, specify the database consistency method.




back the database into operational state.

Figure 99 SAP System Refresh: Database consistency



In the next step, acknowledge the user settings. As it is a SAP system refresh, the users should

already exist.


Figure 100 SAP System Refresh: User and Group Management

In the next screen, enter the password for the schema user and click the “Next” button to

proceed to the next step.

Figure 101 SAP System Refresh: SAPinst Release Configuration

In the next screen, acknowledge or adjust the isolation settings during the network fencing.




Figure 102 SAP System Refresh: Define Allowed Outgoing Connection for System Isolation

Adjust or confirm the PCA settings and click “Next” to continue.

Figure 103 SAP System Refresh: Post Copy Automation

In the next screen, select whether to delete storage volumes at the end of the SAP System

refresh.

Chose the default value to not delete volumes, and click “Next” to get to the next selection

screen.



Figure 104 SAP System Refresh: Storage Volumes

Finally review the settings and start the operation by clicking the “Start System Refresh”

button.



Figure 105 SAP System Refresh: Summary

5.2.7 Execution of SAP System Rename

The execution of the SAP System Rename is very similar to the steps before. Start with the

Provisioning view. The system needs to be enabled for SAP System Rename. Only then the

appropriate button is shown.

Click “Rename system” to start the parameter dialog.



Figure 106 Start SAP System Rename

In the following screen, enter the basic data for the SAP System Rename.

1. Enter and confirm the master password

2. Click “Next” to get to the next selection screen

Figure 107 SAP System Rename: Basic Data of Target System

For convenience, the virtual hostname can be changed on the next screen, as the one from the

system clone will not fit to the naming conventions.

1. Then, select a network

2. Click “Next” to proceed



Figure 108 SAP System Rename: Virtual Hostnames and Networks

SAP Instance number for the renamed system can be specified or, per default, use the same as

the source system. Click “Next” to continue.

Figure 109 SAP System Rename: SAP Instance numbers

On the next panel, the mount points are adjusted to the new SAP System ID. In most cases

you can accept the default names, except if you specifically want to update certain mount

points. Click “Next” to proceed.



Figure 110 SAP System Rename: Mount Data

Select the administrative users for the new SAP System.

If you use LDAP, create users and groups manually in LDAP before, as SAPinst is currently

not able to create LDAP users correctly. However, SAPInst Rename is able to create the SAP

users locally in AIX.

If the users already exist in LDAP or locally in the operating system, then the checkbox “User

already exists” is automatically flagged.

Click “Next” to proceed to the following screen.



Figure 111 SAP System Rename: User and Group Management

In the next step, the settings for the database connect user are specified. For DB2, the connect

user cannot be changed during the System Rename.

1. Specify the password for the connect user

2. Click “Next” to proceed

Figure 112 SAP System Rename: SAPinst Release Configuration



The next screen describes the allowed communications during the network fencing phase.

Ensure that all necessary connections (e.g. to the DNS nameserver) are allowed, but that all

potential dangerous communication is prohibited. Click “Next” to proceed to the following

screen.

Figure 113 SAP System Rename: Define Allowed Outgoing Connections for System Isolation

In the next screen, the SAP target clients for Post Copy Automation execution are specified.

In case you want to disable PCA, remove all the potential clients from the list. If the table is

empty, then the PCA won’t be invoked for any client.

Click “Next” to proceed to the final confirmation screen.

Figure 114 SAP System Rename: Post Copy Automation



In the final selection screen, the parameters can be review before the actual start. Then click

the “Start System Renaming” button to start the execution.

Figure 115 SAP System Rename: Summary



5.2.8 Additional Considerations for SAPInst Rename

See also SAP Note 1619720 - System Rename for SAP Systems based on SAP NetWeaver

for additional information.

In our lab scenario we identified a few issues in the context of SAP system copies. (Some or

all of them may be already fixed with the latest versions of SAP System Rename/SAP Kernel,

but this was not re-tested).

1. The SAP System Copy in SAP LVM procedure doesn’t handle DB2 UDB with

Automatic Storage Management (ASM) correctly.

o During the SAP System Copy, the SAP Kernel executable “brdb6brt” is called

on the source system to generate a parameter file “relocate.template”. The

content of the file is later used on the target system during the “relocatedb”

step: The db2relocatedb tool allows to move the location of an entire DB2

UDB database to the new file system structure (e.g. change of <Source SID>

to <Target SID> in both database and filesystem structure). In case of ASM,

additional parameters are required in the template file:

1. STORAGE_PATH

2. CONT_PATH

Those parameters were not provided from “brdb6brt”

o The ASM directory structure follows the conventions

/db2/<SID>/sapdataX/db2<sid>/NODE0000/<SID>

Not all the <SID> and <sid> entries are adapted by SAP LVM to

the “new” target <SID> / <sid> values of the target system

Resolution: Handled in Event script, see below

2. SAPinst and SAP LVM don’t handle user creation in LDAP correctly

Make sure that the ABAP Connect User on the target host has the

following group sets assigned groups=db<src-sid>mon,db<src-

sid>mnt,db<tgt-sid>mon,db<tgt-sid>mnt


3. SAPSYSTEM RENAME didn’t handle the update to db2nodes.cfg if a “full

qualified hostname” is set as hostname on the target host

If hostname and netname are specified in db2nodes.cfg, only the hostname

is changed. If only the hostname is specified, then it is translated

erroneously


4. An issue with “db2chgpath” was identified in the test environment (DB2 UDB

V9.7 FP5SAP)

Resolution: Fixed in DB2 UDB V9.7 FP6 (APAR IC80849).


http://www-1.ibm.com/support/docview.wss?uid=swg1IC80849



5. Update to AIX 6.1 TL7 SP4 (6100-07-04-1216) introduced a new issue IV22062

on some of the nodes.

Resolution: Install eFIX on top of AIX 6.1 TL7 SP4

root> emgr -l

ID STATE LABEL INSTALL TIME UPDATED BY ABSTRACT

======================================================

1 S IV22062s04 06/11/12 12:42:43 Ifix [email protected]

Item 1.), 2.) and 3.) are addressed by a “Pre-Event” script, which is invoked before SAPInst

rename is started: This is realized via a “Pre-Hook” in SAP LVM. The following outlines how

to implement such a hook. See also SAP Note 1465491 - Provider Implementation Definition

for further details.

5.2.8.1 Create the Script and Register it with the Host Agent

The script is invoked for both SAP Central Instance and Database. It checks, if it is running

for the database, and then identifies the Source SID based on the environment. Target SID is

specified as argument. The script provides the following functionality

Check and eventually create/correct the LDAP users and groups required for the target

system

Handle DB2 UDB ASM

o Update relocate.template to include storage and container pathes in the

template

o Correct directory structure below the /db2/<SID>/sapdataX filesystems

Update db2nodes.cfg (in the directory of the source instance)

When the script is installed, a .conf file needs to be created in the directory

/usr/sap/hostctrl/exe/operations.d so that the Host Agent can start the script.

cat ISICC_PreSAPinst.conf

Name: ISICC_PreSAPinst

Command: /usr/sap/hostctrl/exe/operations.d/IBM/PreSAPinst.ksh

$[SYSTEM_HOST:#required] $[system_sid:#required] $[SERVICE_CLASS:#required]

$[service_name:#required] $[service_type:#required]

Workdir: /usr/sap/hostctrl/exe/operations.d/IBM/traces

ResultConverter: flat

Platform: Unix

In the .conf file, the script and its parameters are defined in the “Command:“ section.

“Workdir:” contains the actual work directory for the script. Make sure that the directory

exists.

Restart the SAP Host Agent to register the script.

5.2.8.2 Define the Provider Implementation Definition

For the Provider Implementation Definition, login to SAP LVM and navigate to

SetupExtensibilityCustom Operations, Hooks, and NotificationsProvider

Implementation Definition. Click the “Add” button to start a new definition.

http://www-304.ibm.com/support/docview.wss?uid=isg1IV22062

mailto:[email protected]




Figure 116 Provider Implementation Definition (I)

On the following screen:

1. Enter name of the hook and select Type “Script Registered with Host Agent”

2. The hostname running a SAP Host Agent where the script was registered

3. Click “Retrieve Registered Scripts”

Figure 117 Provider Implementation Definition (II)



Now, select the registered script from the pick list.

Figure 118 Provider Implementation Definition (III)

The registered script is a custom hook, and it shall be started for an instance (Database, SAP

Central Instance). Checkmark “Instance” in the row for “Hook”. Afterwards, save the new

definition.

Figure 119 Provider Implementation Definition (IV)

5.2.8.3 Define the Custom Hook Definition

The “Custom Hook Definition” then maps the script defined before to the “Pre-SAPInst”

Event. Change to the Custom Hook Definition Tab (SetupExtensibilityCustom

Operations, Hooks, and NotificationsCustom Hooks), and click “Add”.

Figure 120 Custom Hook Definition (I)



On the next panel:

1. Enter a name for the custom hook

2. Select the provider implementation definition defined before from the “Definition

Name” list.

3. Select “Instance” from the “Entity Type” list.

4. Select the operation “SAPinst” from the “Operation” pick list.

Figure 121 Custom Hook Definition (II)

5. Select the hook type “Pre Hook” from the list. The hook is called directly as pre-event

before starting the SAPInst rename step.

Add an “OS Type (static)” constraint in the table at the bottom to ensure the hook runs on

AIX hosts only.

Click “Save” to confirm the changes.

Figure 122 Custom Hook Definition (III)

5.2.9 Destroy a SAP System

During the SAP system destroy action

the storage configuration and all the actual volumes for the systems will be deleted in

the storage system

IP labels, which were created in the DNS server during SAP system creation by SAP

LVM, will be removed from the DNS server once again



The SAP system will be deleted in the SAP LVM repository.

All instances for the system needs to be in “unprepared” state before the “Destroy” action can

be started in SAP LVM. If the instances are still running, then the system needs to be stopped,

and an unprepare operation needs to be started for it first.

5.2.9.1 Stop and Unprepare a SAP System

Navigate to the „Operations“ view and check the status of the system. As a prerequisite to

start the “Destroy” operation, all the instances of the system need to be in state “Unprepared”.

In the example below the system is still active. A mass operation to “Stop and Unprepare” is

selected for both the database and the SAP Central Instance.

Figure 123 System State in Operations View

Click “Execute“ to start the mass operation. The instances get locked for other operations, and

the “Stop and Unprepare” operation is started.

The progress of the current action can be monitored in the “Monitoring” view. On the upper

section of the screen a status and progress overview is given. By selecting an operation from

the list, and choosing the “Steps” sheet the individual steps of the workflow can be monitored.



Figure 124 Monitoring View

Wait until the “Stop and Unprepare“ operation completes for all the instances.

Figure 125 Mass Stop and Unprepare Completed

5.2.9.2 Destroy the SAP System

Navigate to the “Provisioning“ tab and select the system to be destroyed.



Figure 126 Destroy System

Click „Destroy“ System: three additional selection/information steps are processed before the

final destroy operation is started.

During the destroy operation all the storage volumes which were deployed for the system

during an install/copy/clone process are removed on the storage system. In the first step, all

these storage volumes to be deleted later are shown.

Figure 127 System Destroy: Delete Storage Volumes

Click “Next“ to proceed. If SAP LVM is configured for DNS updates and has generated DNS

entries for the system during a SAP LVM deployment process then these entries will get

removed during the destroy operation. This feature was not configured in the test

environment. Instead all IP addresses and corresponding DNS entries were preconfigured in

the DNS before. So in the example case this list is empty.



Figure 128 System Destroy: Host Names

Click “Next” to proceed to the summary screen. The parameters are shown for a final review.

Click “Start System Destroy” to start the actual destroy operation.

Figure 129 System Destroy: Summary



6 Troubleshooting hints and tricks

6.1 Manual cleanup of SAP LVM copy process

Manual cleanup may be required in case one of the SAP System clone, copy, or refresh steps

fails and the SAP LVM task “Destroy partially prepared system” does not work.

SAP Note 1925060 describes the required cleanup steps. Below we list the actions that

should be performed in case of cleanup. Check the above SAP note 1925060 for updates.

6.1.1 Retrieve information details about the process

On SAP source host

Identify the profile directory of the copy process. The directory is located within FCM

Directory as configured in SAP LVM Storage Manager configuration and begins

with "SAPLVM_<TARGET_HOST>_<SOURCE_HOST>_<STORAGE_ID>_".

On SAP source host

Go to the directory found in the previous step and check if "backup.log" is available.

o Check for lines with the following pattern:

"#SAVED <PREFIX>__<BACKUP_ID> <DIRECTORY>"

(e.g. "#SAVED SAP0__A0HCJ22U1M /usr/sap/L43")

Note the Backup ID.

o In case the last line of backup.log shows "FMM0024I Return code is: 2.", the

backup id is not relevant and the last step of this cleanup procedure can be

skipped.

o In case backup.log is not available, the last step of this cleanup procedure can

be skipped.

In SAP LVM UIMonitoringLogs

Open SAP LVM logs of PreProcessCloneVolumes step and note target volume names

from context.

In SAP LVM UIConfiguration:

o Go to SAP LVM mount configuration for each of the SAP target system

instances.

o Note mount points that should become available on SAP target host after

successful process.

o Check if the values for export path of the mount points are not empty. If the

export path contain comma separated IDs, note them.

6.1.2 Cleanup hosts

Stop SAP instances and all instance agents

Stop the SAP Host Agent

Clean up FCM process on SAP copy central management host

o Get corresponding "acsd" process and kill it. "ps -ef | grep acsd" will show a

process that uses profile directory identified in first step. The directory name

will end with "_acsd". (e.g. "kill -9 12345")

o Delete or move profile directory identified in first step. (e.g. "mv

/home/sapadm/acs/SAPLVM_* /usr/sap/hostctrl/exe/operations.d/IBM/traces")

Clean up FCM process on SAP copy source host

o Get corresponding "acsgen" process and kill it. "ps -ef | grep acsgen" will

show a process that uses profile directory identified in first step.



o Delete or move profile directory identified in first step. (e.g. "mv

/home/sapadm/acs/SAPLVM_* /usr/sap/hostctrl/exe/operations.d/IBM/traces")

Clean up FCM process on SAP copy target host

o Get corresponding "acsgen" process and kill it. "ps -ef | grep acsgen" will

show a process that uses profile directory identified in first step.

o Delete or move profile directory we found previously. (e.g. "mv

/home/sapadm/acs/SAPLVM_* /usr/sap/hostctrl/exe/operations.d/IBM/traces"

Clean up firewall rules on SAP copy target host

o Check if SAP LVM related firewall rules are available. SAP LVM related

firewall rules begin with "SLM" in rule description.

"lsfilt -v4 -O" will show all currently active firewall rules.

Delete all SAP LVM related firewall rules

Delete the "deny" rules first, otherwise the SAP target host may become isolated.

"rmfilt -v4 -n xxx" will delete the firewall rule with ID xxx.

Activate updated firewall rule list afterwards

Use "mkfilt -v4 -u" to activate the changes.

Clean up storage file systems on SAP copy target host

o Check with "df -g" if any of file systems retrieved previously from SAP LVM

mount configuration of SAP target system instances are mounted.

o Unmount all of these file systems (e.g. "umount -f /usr/sap/CPY").

o Get available logical volume group names with "lsvg".

o Identify with "lsvg -l <VGNAME>" all the logical volume groups that include

file systems retrieved previously from SAP LVM mount configuration of SAP

target system instances.

o Execute "varyoffvg <VGNAME>" and "exportvg <VGNAME>".

Prerequisite: all the file systems listed in logical volume groups with "lsvg -l

<VGNAME>" are closed.

o Execute "lspv" and get all the physical disks on the system.

o Remove all the physical disks that do not belong to any logical volume groups

from the operating system by executing "rmdev -d -l hdiskX"

Clean-up /usr/sap/sapservices

Remove all entries set by SAP LVM on the target host

o /etc/hosts

o /etc/services

Restart SAP Host Agent

6.1.3 Cleanup storage

Identify storage volumes and/or snapshots that should be removed

In order to identify storage volumes / snapshots use target volume names and/or

Backup ID and/or IDs from export path in SAP LVM mount configuration retrieved

previously. Target volume names correspond to storage volume / snapshot names,

Backup ID will be a part of snapshot name (on XIV) and IDs from export path in SAP

LVM mount configuration correspond to storage volume / snapshot WWNs (on XIV,

V7000 or on SVC)

Ensure that the storage volumes / snapshots are NOT masked

This process is very specific and depends on the storage subsystem or storage cluster.

Below we describe only actions that must be performed but not the exact process

itself.



o Check if storage volumes / snapshots are attached to host alias / port IDs that

represent SAP target host.

o Ensure that you completed clean up storage file systems procedure on SAP

copy target host. Otherwise, you will need to restart SAP target host because of

AIX volume group errors.

o Detach storage volumes / snapshots from host alias / port IDs that represent

SAP target host.

On V7000 storage system or on SVC cluster: remove consistency groups and

FlashCopy relations that include storage volumes to be deleted:

o Remove FlashCopy relations from consistency group

o Remove consistency group

o Bring up FlashCopy relations into a final state (e.g. stop them if "Stop" action

is available)

o Delete FlashCopy relation. Set optional flag "Delete the FlashCopy mapping

even when the data on the target volume is inconsistent, or if the target volume

has other dependencies."

Delete storage volumes and/or snapshots.

6.2 Delete the System in SAP LVM

Go to SAP LVM UIConfigurationSystems

Select the appropriate system, and press “Remove Selected Instances and Systems” (Figure

130).

Confirm the deletion in the following conformation dialog.

Figure 130 Remove System in SAP LVM



7 Resources

7.1 IBM Documentation

7.1.1 Whitepapers

Flyer “Cloud-enabled SAP application management”

IBM Cloud Solution for SAP – Integrating IBM Infrastructure with SAP Landscape

Virtualization Management

Introduction to an Integral IBM / SAP Cloud Solution for SAP Landscape Management

IBM entry cloud configuration for SAP solutions on Power Basic Concepts and High Level

Design

7.1.2 Redbooks / Technotes

Deploying Cloud Components on POWER


IBM PureFlex System and IBM Flex System Products and Technology

IBM Flex System p260 and p460 Compute Node

7.1.3 Product Documentation

Requirements and support for AIX using Network Installation Manager (NIM)

Requirements and support for AIX, IBM i, and Linux using storage copy services (SCS)

7.1.4 iRAM

IBM Systems Director VMControl Implementation Perform Guide

7.2 SAP Documentation and SAP Notes

7.2.1 SAP Documentation

SAP NetWeaver Landscape Virtualization Management

SAP Community Network (SCN) Virtualization and Cloud SAP Help Portal (SAP LVM)

SAP NetWeaver Landscape Virtualization Management, Enterprise

SAP NetWeaver Landscape Virtualization Management at a Glance

7.2.2 SAP Notes

http://public.dhe.ibm.com/common/ssi/ecm/en/sps03044deen/SPS03044DEEN.PDF

https://scn.sap.com/docs/DOC-24822

https://scn.sap.com/docs/DOC-24822

http://www.ibm.com/support/techdocs/atsmastr.nsf/WebIndex/PRS4803

ftp://submit.boulder.ibm.com/sales/ssi/ecm/en/pol03119usen/POL03119USEN.PDF

ftp://submit.boulder.ibm.com/sales/ssi/ecm/en/pol03119usen/POL03119USEN.PDF

http://www.redbooks.ibm.com/redbooks/pdfs/sg248009.pdf

http://www.redbooks.ibm.com/technotes/tips0862.pdf

http://www.redbooks.ibm.com/redbooks/pdfs/sg247984.pdf

http://www.redbooks.ibm.com/technotes/tips0880.pdf

http://publib.boulder.ibm.com/infocenter/director/pubs/topic/com.ibm.director.vim.helps.doc/fsd0_vim_r_power_virtualization_using_NIM.html

http://publib.boulder.ibm.com/infocenter/director/pubs/topic/com.ibm.director.vim.helps.doc/fsd0_vim_r_sb_aix_on_power.html

https://w3-03.ibm.com/tools/cm/iram/artifact/CFAA94C6-4CF7-6FC3-9D66-C0ACB1AAAA81/1.0/Additional%20Delivery%20Assets/IBM%20Systems%20Director%20VMControl%20Implementation%20Perform%20Guide_v02.rar

http://www.sap.com/solutions/technology/cloud/virtualization/index.epx

http://scn.sap.com/community/it-management/virtualization

http://help.sap.com/nwlvm

http://help.sap.com/saphelp_lvm10enterprise/helpdata/en/4d/f88a8f418c5059e10000000a42189c/frameset.htm

http://scn.sap.com/docs/DOC-25771



SAP Note Title / Description

Note 962955 Use of virtual TCP/IP host names

Note 1292836 Logfiles of the SAP Hostagent

Note 1438774 New profile parameter system/uuid and system/description

Note 1465491 Provider Implementation Definition

Note 1527538 SAP NetWeaver Landscape Virtualization Management 1.0

Note 1572841 Setup of name server update of SAP LVM

Note 1589145 Task Manager for Technical Configuration

Note 1589175 System Copy: Task Content for Task Manager

Note 1614266 System Copy: Post Copy Automation (PCA) / LVM 1.0

Note 1619720 System Rename for SAP Systems based on SAP NetWeaver

Note 1644520 Integration of IBM Components into SAP LVM

Note 1728222 Configuration of IBM Systems Director VMControl for SAP LVM

Note 1728293 Configuration of IBM Power HMC for SAP LVM

Table 19 List of SAP Notes
















COPYRIGHT LICENSE

If this information contains sample application programs in source language, which illustrate

programming techniques on various operating platforms, you may copy, modify, and

distribute these sample programs in any form without payment to IBM, for the purposes of

developing, using, marketing or distributing application programs conforming to the

application programming interface for the operating platform for which the sample programs

are written. These examples have not been thoroughly tested under all conditions. IBM,

therefore, cannot guarantee or imply reliability, serviceability, or function of these programs.

The information contained in this document has not been submitted to any formal IBM test

and is provided "AS IS" with no warranties or guarantees either expressed or implied.

All examples cited or described in this document are presented as illustrations of the manner

in which some IBM products can be used and the results that may be achieved. Actual

environmental costs and performance characteristics will vary depending on individual client

configurations and conditions.

IBM is not responsible for printing errors in this document that result in pricing or

information inaccuracies.

A full list of U.S. trademarks owned by IBM may be found at:

http://www.ibm.com/legal/copytrade.shtml

A full list of trademarks owned by SAP may be found at:

http://www.sap.com/company/legal/copyright/trademark.epx

SAP and other SAP products and services mentioned herein, as well as their respective logos,

are trademarks or registered trademarks of SAP AG in Germany and in several other countries

all over the world.

UNIX is a registered trademark in the United States, other countries or both.

Linux is a registered trademark of Linus Torvalds in the United States, other countries or

both.

Microsoft, Windows, Windows NT and the Windows logo are trademarks of Microsoft

Corporation in the United States, other countries or both.

http://www.ibm.com/legal/copytrade.shtml

http://www.sap.com/company/legal/copyright/trademark.epx

integration of ibm infrastructure and sap lvm v2.0

Documents

ibm practitioners

ibm power systems

ibm sap technical brief

sap infrastructure

ibm systems director

ibm power environment

ibm technical sales

ibm infrastructure specifics