flexframe agents guide -...

FlexFrame®

Version 5.3A / 1.0A

FlexFrame Agents Installation and Administration

Edition 06.05.14 Document Version 1.4

Fujitsu Limited

© Copyright Fujitsu Technology Solutions 2013

FlexFrame® and PRIMERGY™ are trademarks or registered trademarks of Fujitsu Limited in Japan

and other countries.

myAMC® is a registered trademark of Atos in Germany and in several other countries.

SAP® and NetWeaver™ are trademarks or registered trademarks of SAP AG in Germany and in

several other countries

Linux® is a registered trademark of Linus Torvalds

SUSE® Linux is a registered trademark of Novell, Inc., in the United States and other countries

Oracle™ and Java™ are trademarks of ORACLE Corporation and/or its affiliates

Intel® and PXE

® are registered trademarks of Intel Corporation in the United States and other coun-

tries

MaxDB® is a registered trademark of MySQL AB, Sweden

MySQL® is a registered trademark of MySQL AB, Sweden

postgreSQL® is a registered trademark of PostgreSQL global development group

NetApp® and the Network Appliance® logo are registered trademarks and Network Appli-

ance™ and Data ONTAP™ are trademarks of NetApp, Inc. in the U.S. and other coun-

tries.

EMC®, CLARiiON

®, Symmetrix

® and Celerra™ are trademarks or registered trademarks of EMC

Corporation in the United States and other countries

VMware®, ESX

®, ESXi, VMware vCenter, VMware vSphere are registered trademarks or trademarks

of VMware, Inc. in the United States and/or other jurisdictions.

Ethernet® is a registered trademark of XEROX, Inc., Digital Equipment Corporation and Intel Corpo-

ration

Windows® and Word

® are registered trademarks of Microsoft Corporation

All other hardware and software names used are trademarks of their respective companies.

All rights, including rights of translation, reproduction by printing, copying or similar methods, in part

or in whole, are reserved.

Offenders will be liable for damages.

All rights, including rights created by patent grant or registration of a utility model or design, are reserved. Delivery subject to availability. Right of technical modification reserved.

3 FlexFrame Agents Guide


Contents

General product information and naming .................................................................... 11

1 Introduction ................................................................................................... 13 1.1 FlexFrame Autonomy ...................................................................................... 14 1.2 Additional Documentation ............................................................................... 14 1.3 Target Group ................................................................................................... 15 1.4 Notational Conventions ................................................................................... 15 1.5 Document History ............................................................................................ 16 1.6 Changes and Enhancements in this Version ................................................... 17 1.7 Related Documents ......................................................................................... 18

2 First Steps ...................................................................................................... 19 2.1 General Information ......................................................................................... 19 2.2 Installation and Startup .................................................................................... 20 2.2.1 The FlexFrame Solution .................................................................................. 20 2.2.2 Installation ....................................................................................................... 21 2.2.3 Configuration ................................................................................................... 33 2.2.4 Starting and Stopping ...................................................................................... 34 2.2.5 FA-Agents in different FlexFrame Versions ..................................................... 35 2.2.6 Configuration of generic .................................................................................. 41 2.3 FA WebInterface ............................................................................................. 43 2.3.1 Function .......................................................................................................... 43 2.3.2 Installation ....................................................................................................... 43 2.3.3 Configuration ................................................................................................... 43 2.3.4 Starting and Stopping ...................................................................................... 43 2.3.5 myAMC.FA WebGui and database ................................................................. 43 2.4 DomainManager .............................................................................................. 46 2.4.1 Database connection ....................................................................................... 46 2.5 Messenger....................................................................................................... 48 2.5.1 Database connection ....................................................................................... 48 2.6 myAMC.FA version management .................................................................... 50 2.6.1 Lifecycle process ............................................................................................. 50 2.6.2 Lifecycle management files ............................................................................. 51 2.7 Usage of Pool-Master ...................................................................................... 57 2.7.1 Create Pool-Master from existing pool ............................................................ 58 2.7.2 Create/Update pool from Pool-Master ............................................................. 58 2.7.3 Usage of ‘fa_MasterPoolMgr.sh’ ..................................................................... 60 2.7.4 Automation of steps during pool creation ........................................................ 60

3 Introduction ................................................................................................... 63 3.1 FlexFrame Autonomy Architecture .................................................................. 63

Contents

FlexFrame Agents Guide 6

3.1.1 FlexFrame Autonomy Visualization and Reporting .......................................... 65 3.1.2 FlexFrame Autonomy virtual server and hypervisor nodes .............................. 66 3.2 Pool Creation and Grouping ............................................................................ 68 3.2.1 Virtual FlexFrame Autonomy Pools ................................................................. 68 3.2.2 Grouping .......................................................................................................... 71 3.2.3 Pool Types (Pool Independent Spare) ............................................................. 71 3.3 Service Classes ............................................................................................... 75 3.3.1 Service Priority ................................................................................................ 75 3.3.2 Service Power Value ....................................................................................... 75 3.3.3 Class Creation Rules ....................................................................................... 76 3.3.4 Testament Types ............................................................................................. 76 3.4 FA Configuration, Work and Log Files ............................................................. 77 3.5 Systems ........................................................................................................... 78 3.5.1 Service Types .................................................................................................. 78 3.6 Generic Services ............................................................................................. 81 3.6.1 Service State Model ........................................................................................ 81 3.6.2 Service Detection Model .................................................................................. 82 3.6.3 Service Reaction Model ................................................................................... 82 3.7 FlexFrame Performance and Accounting Option ............................................ 83 3.7.1 Performance Option ........................................................................................ 84 3.7.2 Accounting Option ........................................................................................... 85 3.7.3 Billing ............................................................................................................... 86 3.8 Parametricable Value Observation and Service Level Profile ......................... 88 3.8.1 Monitoring of file system and directoy utilization .............................................. 88 3.8.2 Performance Profile, Capacity Profile, Accounting Profile ............................... 88 3.9 FlexFrame File System Utilization and File Space Accounting Option ............ 89 3.9.1 File System Detector and Directory Detector ................................................... 92 3.10 Domain Manager for Performance and File System Collections ..................... 93 3.10.1 Data collection ................................................................................................. 93 3.10.2 Data retentiontime and compression ............................................................... 94

4 FlexFrame Autonomy .................................................................................... 95 4.1 FlexFrame Autonomy Reactions ..................................................................... 96 4.1.1 Restart ............................................................................................................. 96 4.1.2 Reboot ............................................................................................................. 96 4.1.3 SwitchOver ...................................................................................................... 96 4.1.4 Maintenance .................................................................................................... 99 4.2 Self-Repair Strategies ..................................................................................... 99 4.2.1 Self-Repair in the Event of a Service Failure ................................................... 99 4.2.2 Self-Repair in the Event of a Node Failure .................................................... 100 4.2.3 Takeover by a Spare Node (TakeOver) ......................................................... 100 4.2.4 Multi Node Failure ......................................................................................... 100 4.3 Escalation with partner service ...................................................................... 102 4.3.1 SCS / ASCS in conjunction with ERS ............................................................ 102 4.3.2 Service with assigned SMD-service .............................................................. 103

Contents


4.4 Takeover Rules ............................................................................................. 105 4.4.1 Overview ....................................................................................................... 105 4.4.2 TakeOver Strategy ........................................................................................ 105 4.4.3 TakeOver Rule .............................................................................................. 106 4.4.4 Spare Nodes from the Adminpool (pool-independent spare node)................ 113 4.5 Management of Performance, File System and Accounting .......................... 121 4.5.1 Performance and Accounting Management .................................................. 121 4.5.2 File System Utilization ................................................................................... 121 4.5.3 Monitoring Cycles .......................................................................................... 121 4.6 Operating Mode ............................................................................................. 122 4.6.1 Event Mode ................................................................................................... 122 4.6.2 Local Reaction Mode ..................................................................................... 122 4.6.3 Central Reaction Mode .................................................................................. 122 4.7 Autonomous Operation of a FlexFrame Infrastructure ................................. 124 4.7.1 FlexFrame Autonomy and Fujitsu FlexFrame Scripts .................................... 124 4.8 FlexFrame Autonomy and User Interactions ................................................. 125 4.8.1 myAMC.FA_Agents: Starting/Stopping/Status .............................................. 126 4.8.2 Starting/Stopping an SAP Instance ............................................................... 128 4.8.3 NoWatch/Watch an SAP Instance ................................................................. 130 4.9 Use cases...................................................................................................... 131 4.9.1 General ......................................................................................................... 131 4.9.2 Semi-autonomous operation ......................................................................... 132 4.9.3 Autonomy for Application Instances .............................................................. 133 4.9.4 Management of Performance, Capacity and Accounting .............................. 135 4.9.5 File System Utilization ................................................................................... 135 4.9.6 High Availability and Autonomy Rules ........................................................... 136 4.10 FA Work and Log Files .................................................................................. 146 4.11 Migration of FA Agent Versions on Pool Level .............................................. 147 4.12 The FA Migration Tool ................................................................................... 151 4.12.1 Pool Mode ..................................................................................................... 151 4.12.2 File Mode....................................................................................................... 151 4.12.3 Usage ............................................................................................................ 152 4.13 Command Line Interface fa_list_services.sh ................................................. 154 4.14 Command Execution at All Nodes of a Pool / ALL_POOLS .......................... 155

5 WebInterface ................................................................................................ 157 5.1 Installation / Configuration ............................................................................. 157 5.1.1 Prerequisites ................................................................................................. 157 5.1.2 Installation ..................................................................................................... 157 5.1.3 Configuration ................................................................................................. 157 5.2 Visualization .................................................................................................. 162 5.2.1 Starting the WebInterface / Access via Web Browser ................................... 162 5.2.2 Login ............................................................................................................. 162 5.2.3 Overview of Elements ................................................................................... 163 5.2.4 Pool / Group Tree .......................................................................................... 164

Contents


5.2.5 Status Display ................................................................................................ 170 5.2.6 Message Display ........................................................................................... 176 5.2.7 Configuration of FlexFrame Autonomy with the Webinterface ....................... 181 5.2.8 Special Signals .............................................................................................. 191 5.3 Interaction ...................................................................................................... 192 5.3.1 Updates ......................................................................................................... 192 5.4 Info and Help ................................................................................................. 194 5.5 FlexFrame Performance and Accounting Plug-in .......................................... 195 5.6 FlexFrame Reporting Plug-in ......................................................................... 195 5.7 Customization of Icons in the FA WebGUI .................................................... 196

6 FlexFrame Autonomy Power Shutdown Concept ..................................... 197 6.1 General .......................................................................................................... 197 6.2 Power Shutdown Architecture ....................................................................... 198 6.3 Basics ............................................................................................................ 199 6.3.1 Powershutdown methods for use with FA generic shutdown facility .............. 199 6.4 Power Shutdown Configuration ..................................................................... 201 6.4.1 Switchover Control Parameters ..................................................................... 202 6.4.2 User, Password and Community ................................................................... 204 6.4.3 Management Blades ...................................................................................... 207 6.4.4 Application Nodes .......................................................................................... 209 6.4.5 Default Shutdown Mode ................................................................................ 214 6.4.6 Management blades with SNMP-community different from default ............... 214 6.5 Possible Errormessages with old Configuration ............................................ 215

7 Parameter Reference ................................................................................... 217 7.1 FA Agents ...................................................................................................... 217 7.1.1 FA Agent Configuration Files ......................................................................... 217 7.2 SNMP Traps .................................................................................................. 218 7.2.1 General .......................................................................................................... 218 7.2.2 Structure ........................................................................................................ 218 7.2.3 Default Parameter File ................................................................................... 220 7.3 Pooling and Grouping .................................................................................... 221 7.3.1 Pooling........................................................................................................... 221 7.3.2 Grouping ........................................................................................................ 221 7.3.3 Default Parameter File ................................................................................... 226 7.4 Service Classes ............................................................................................. 245 7.4.1 Service Priority .............................................................................................. 245 7.4.2 Service Power Value ..................................................................................... 245 7.4.3 Class Creation Rules ..................................................................................... 245 7.4.4 Example......................................................................................................... 245 7.5 FlexFrame Autonomy .................................................................................... 246 7.5.1 General Parameters ...................................................................................... 247 7.5.2 Parameters for the Performance and Accounting Option .............................. 249 7.5.3 Parameters for the File System Utilization ..................................................... 249

Contents


7.5.4 Configuration of the Spaceutil Functionality .................................................. 249 7.5.5 Parameters for the Admin Pool Option .......................................................... 259 7.5.6 Parameters for ServicePing........................................................................... 260 7.5.7 Node-related Parameters .............................................................................. 260 7.5.8 Service-related Parameters ........................................................................... 261 7.5.9 Parameters for the Definition of a Generic Service ....................................... 263 7.5.10 Path Configuration ......................................................................................... 268 7.5.11 Shutdown Configuration ................................................................................ 270 7.5.12 Default Parameter File .................................................................................. 271

<?xml version="1.0" encoding="ISO-8859-1"?> ........................................... 272 7.5.13 FA FrameAgent ............................................................................................. 303 7.5.14 Adjustment of the FA WebGUI ...................................................................... 303 7.5.15 FA Webservices and Hypervisor Monitor configuration ................................. 304 7.6 FA DomainManager ...................................................................................... 305 7.6.1 Common ........................................................................................................ 305 7.6.2 Configuration ................................................................................................. 305

8 BlackBoard .................................................................................................. 311 8.1 General ......................................................................................................... 311 8.2 Implementation .............................................................................................. 311 8.3 Generating BlackBoard Commands .............................................................. 314 8.3.1 WebInterface ................................................................................................. 314 8.3.2 Interactive ...................................................................................................... 315 8.4 Commands .................................................................................................... 315

9 FlexFrame Autonomous Agent Traps ....................................................... 317 9.1 General ......................................................................................................... 317 9.2 Structur of FlexFrame Autonomy Traps ........................................................ 318 9.3 Overview of the FlexFrame Autonomy SNMP Traps ..................................... 321

10 FA Troubleshooting .................................................................................... 337 10.1 General ......................................................................................................... 337 10.2 Incident Diagnosis..................................................................................... 337 10.3 FA Work and Log Files .................................................................................. 341 10.3.1 General ......................................................................................................... 341 10.3.2 Overview, Principal Directories, Files ............................................................ 341 10.3.3 Collecting Diagnostic Information for Support Assistance ............................. 347 10.3.4 Selected Files ................................................................................................ 349 10.3.5 FA Autonomy Diagnostic Tool ....................................................................... 350 10.3.6 Collecting Diagnostic Information for Support Assistance ............................. 351

11 Abbreviations .............................................................................................. 353

Contents


12 Glossary ....................................................................................................... 357

13 Index ............................................................................................................. 363


General product information and naming

This manual describes the functional concepts and the application scenarios of the Flex-

Frame® Agents.

This solution is offered in two characteristics:

the FlexFrame® for SAP

® V5.3A solution

This is the successor of the well-known solution FlexFrame® for SAP

®

which is in the market for years with its latest version V5.2A.

the FlexFrame® Orchestrator V1.0A solution

This is a new approach to offer enhanced functionality and features

(e.g. support of SAP HANA, support of KVM as virtual machine) step by

step and become more and more independent from certain hardware

and software components. These additional new features compared to

the traditional FlexFrame for SAP features are available by special li-

censes only.

This documentation is for both characteristics:

All descriptions regarding FlexFrame Orchestrator have a corresponding notifi-

cation within their chapters.

If there is no notification regarding FlexFrame Orchestrator the described func-

tionality is valid for both characteristics.

The name FlexFrame® is a generic term for both „FlexFrame

® for SAP

®“ (FF4S) and

„FlexFrame® Orchestrator“ (FFO).

The trademark myAMC® is not be used any more. The brand myAMC

® will be replaced

by the brand FlexFrame®, e.g. instead of myAMC Accounting we now speak about Flex-

Frame Accounting.

The abbreviation FA for FlexFrame Autonomy now denotes the FlexFrame Agents

(Application Agents, Control Agents and the Frame Agent). The FA Suite now denotes

the FlexFrame Agents Suite.

These changes have not yet been made in this manual, but will be carried out in the next

edition.


1 Introduction

For many companies, applications such as SAP® today provide the basis for handling all

important business processes. Failure of these components therefore results in consider-

able costs. Nowadays companies must be able to react very rapidly to changing market

and organizational demands, which also means that it must be possible to adapt the

capacity of existing IT resources very quickly to the changing requirements.

The myAMC components for monitoring the availability and utilization of IT systems with

their intelligent automated facility for responding to system failures are the answer to

these demands. FlexFrame® Autonomy complements the powerful monitoring and man-

agement functions of myAMC with functions which permit the autonomous operation of a

distributed applications environment. These functions reduce the number of manual inter-

ventions and make the operation of your business critical applications more efficient.

FlexFrame offers a flexible hardware architecture which can be adapted to altered re-

quirements and, together with management components, permits highly available opera-

tion of this infrastructure. Partial outages are automatically repaired or compensated for.

FlexFrame Autonomy is an integral component of every FlexFrame solution and provides

the functions for implementing operation with considerably reduced operator interventions

through the built-in autonomy functions, right up to a high-availability solution.

This manual describes the functional concepts and the application scenarios for Flex-

Frame Autonomy.

FlexFrame Autonomy for distributed database instances, SAP central instances and SAP

application instances

FlexFrame Autonomy supports SAP, SAPDB and Oracle instances

Status monitoring, restart, reboot or switchover of instances

Introduction


1.1 FlexFrame Autonomy

The Application Management Center myAMC is a solution for monitoring and managing IT

infrastructures. The FlexFrame Autonomy component myAMC.FA substantially extends

the range of functions. In addition to monitoring, this component also provides the option

of implementing automatic restoration of failed services autonomously. These self-repair

mechanisms are not just effective locally for one system, however; they also permit a

failed service to be moved automatically to another resource which, in line with a defined

rule for operation, is suitable for operating the service.

This function permits a considerable reduction in the number of manual interventions by

an administrator. Availability is increased, and the costs for operating a complex applica-

tions environment are reduced.

For this functionality, myAMC.FA uses its agents and management components to detect,

collect and analyze the information. Autonomous functions can be configured for various

tasks and requirements by combining different detectors and manager components and

by defining and selecting the reaction and decision rules. In conjunction with the powerful

myAMC GUI, the entire infrastructure can be presented in a straightforward manner in an

IT cockpit.

1.2 Additional Documentation

Further application options for other myAMC management components are described in

special documents.

Use of the Messenger for editing and forwarding myAMC.FA messages is described in the

documentation “FlexFrame Messenger Concept and Usage". Use of the LogAgent for

forwarding of log-entries is described in the documentation “FlexFrame LogAgent Con-

cept and Usage “.

Introduction


1.3 Target Group

This documentation is intended to support both users of FlexFrame Autonomy and ad-

ministrators who wish to integrate this solution in an enterprise IT management solution.

1.4 Notational Conventions

The following conventions are used in this manual:

Additional information that should be observed.

Warning that must be observed.

fixed font Names of paths, files, commands, and system output.

<fixed font> Names of variables.

fixed font User input in command examples

(if applicable using <> with variables)

Introduction


1.5 Document History

Document Version Changes Date

1.0 FA Agents Version 9.0 2010-08-06

1.1 Update 2010-12-08

1.2 Update 2011-02-11

1.3 Update 2011-04-12

1.4 Update 2011-05-xx

1.5 Update 2011-06-28

1.6 Update 2011-11-04

1.7 Update 2012-04-05

1.8 Update for FF4S 5.2 2012-08-03

1.9 Update for FF4S 5.2 2012-08-24

1.10 Update for FF4S 5.2 2012-09-28

1.11 Update for FF4S 5.2 2012-10-31

1.12 Update for FF4S 5.2A00-018 2013-02-11

1.13 Update for FF4S 5.2A00-018 2013-02-27

1.0 First Edition for FlexFrame Orchestrator 2013-10-11

1.1 Update for FlexFrame Orchestrator 2014-02-28




Introduction


1.6 Changes and Enhancements in this Version

Automation of pool creation with a master-pool configuration

BOBJ 4.1 as new service variant additional to BOBJ 3.2 and BOBJ 4.0

Usage of KVM as additional virtualization platform

SAP HANA as additional monitored SAP service

Configuration of the service priority with the myAMC.FA WebGui

Separate sets of rules for groups within a pool can be configured

Persistent “NoWatch”

EzSetup-parameter for force installation of packages, even if same version al-

ready exists

Introduction


1.7 Related Documents

FlexFrame® – Administration and Operation

FlexFrame® – HW Characteristics Quickguides

FlexFrame® – Installation and Configuration LVM 1.0 Standard Edition

FlexFrame® – Installation Guide for SAP Solutions

FlexFrame® – Installation of a FlexFrame Environment

FlexFrame® – Management Tool

FlexFrame®

– Agents Installation and Administration

FlexFrame®

– Messenger Concept and Usage

FlexFrame®

– LogAgent Concept and Usage

FlexFrame® – Network Design and Configuration Guide

FlexFrame® – Security Guide

FlexFrame® – Technical White Paper

FlexFrame® – Upgrading FlexFrame 5.1A or 5.2A to 5.3A/1.0A

ServerView Documentation

SUSE Linux Enterprise Server Documentation


2 First Steps

2.1 General Information

This chapter describes how you start and stop the FlexFrame Autonomy components. It

also describes how FlexFrame Autonomy is installed and its basic configuration.

FlexFrame Autonomy provides a comprehensive, flexible and scalable solution for setting

up semi-autonomous IT processes. Its functionality falls into these subareas:

FA_AppAgents: FlexFrame Autonomy Application Agents for monitoring, checking and

controlling instances

FA CtrlAgent / FA FrameAgent: FlexFrame Autonomy Control Agent / Frame Agent for

monitoring, checking and controlling Application Nodes with a separate Control

Node.

FA_WebInterface: A component for displaying the active services on a web front-end.

FA_Messenger: A component for receiving and forwarding messages.

FA_LogAgent: A component for controlling logfiles.

FA_DomainManager: A component for performance- accounting- and fileutilisation op-

tion.

FA_Suite: common component for example version management.

To monitor instances, the FA_AppAgent supplies cyclical information on the availability of

an instance in a definable rhythm. For this purpose it is necessary that the FA_AppAgent

is active on every node.

myAMC.FA_Messenger is used to forward information on faults and autonomous reac-

tions to the outside. This messaging component of the myAMC family should be operated

on the Control Node.

First Steps


2.2 Installation and Startup

2.2.1 The FlexFrame Solution

The FlexFrame Autonomy solution was conceived and developed especially for the

“FlexFrame® for SAP

® “ solution from Fujitsu. Consequently the FlexFrame solution with

the components Shared OS, Virtualized SAP Application and NetApp Storage on the

target computers is a prerequisite for the procedure described in the following.

Further details on FlexFrame configurations can be found in the FlexFrame manual “In-

stallation of a FlexFrame Environment“. Use of FlexFrame Autonomy on other Linux

architectures (e.g. standalone systems or for monitoring processes which do not belong

to SAP R/3) is not described in this manual and is not supported.

The following prerequisites are thus particularly important:

Server architecture with IP storage (NetApp Filer) and client, server and storage LANs.

Paths for read-only and read/write Root images.

SAP start scripts from Fujitsu

Operating system SUSE Linux Enterprise Server (SLES)

FA Agents are installed in a directory on the storage system which is reachable and

available to all nodes in accordance with the FlexFrame rules for jointly used programs.

Programs are always accessed and installed via a Control Node.

The FA Agents are installed using RPM packages. Normally the agents are stored in the

directories /opt/myAMC/FA_AppAgent, /opt/myAMC/FA_CtrlAgent and

/opt/local/myAMC/FA_FrameAgent.

The /opt/myAMC directory is located in a FlexFrame environment on the Filer and is

available from every Application Node and Control Node.

Multiple FlexFrame Autonomy versions can be installed simultaneously. Installation, con-

figuration and activation of a version are three separate activities.

Installation, parameterization and configuration of a new version can thus be performed

during ongoing operation. Only when all preparations have been completed is the active

version deactivated and the new version activated.

Deactivation and activation of a version always takes place on a pool-specific basis. In

this way new agent versions can, for example, first be activated in a pool with test sys-

tems.

First Steps


2.2.2 Installation

2.2.2.1 Requirements

In the case of a FlexFrame standard installation, new software components are installed

via one of the Control Nodes. The FlexFrame Autonomy software is contained in the /opt/myAMC resp. /opt/local/myAMC directory. Ensure that all servers (Control

Nodes and Application Nodes) use the same directories. FlexFrame Autonomy is thus

also installed in a tree on a Filer (NFS share).

The NFS file systems used have to support NFS file locking.

control1:/opt/myAMC # mount

filer1_qa:/vol/volFF/FlexFrame/myAMC on /FlexFrame/myAMC type nfs

(rw,nfsvers=3,intr,noac,wsize=32768,rsize=32768,addr=172.16.1.204)

filer1_qa:/vol/volFF/FlexFrame/scripts on /FlexFrame/scripts type nfs

(rw,nfsvers=3,intr,nolock,noac,wsize=32768,rsize=32768,addr=172.16.1.204)

control1:/opt/myAMC # ls -al /opt/myAMC

lrwxrwxrwx 1 root root 16 Dec 2 18:35 /opt/myAMC ->

/FlexFrame/myAMC

2.2.2.2 Installation Packages

The following packages must be installed:

myAMC.FlexFrame Autonomy Suite; the installation package for this is called

myAMC.FA_Suite-<X.Y-Z>.i386.rpm

myAMC.FlexFrame Autonomy Application Agent; the installation package for this is called

myAMC.FA_AppAgent-<X.Y-Z>.i386.rpm

myAMC.FlexFrame Autonomy Control Agent; the installation package for this is called

myAMC.FA_CtrlAgent-<X.Y-Z>.i386.rpm

myAMC.FlexFrame Autonomy Frame Agent; the installation package for this is called

myAMC.FA_FrameAgent-<X.Y-Z>.i386.rpm

myAMC.FlexFrame Autonomy WebInterface; the installation package for this is called

myAMC.FA_WebGui-<X.Y-Z>.i386.rpm

myAMC.FlexFrame Autonomy Messenger; the installation package for this is called

myAMC.FA_Messenger-<X.Y-Z>.i386.rpm

myAMC.FlexFrame Autonomy LogAgent; the installation package for this is called

myAMC.FA_LogAgent-<X.Y-Z>.i386.rpm

First Steps


myAMC.DomainManager (optional, e.g.for the performance and accounting option), The

installation package for this is called

myAMC.FA_DomainManager-X.Y-Z.i386.rpm

where X.Y-Z stands for the version number.

2.2.2.3 rpm-Installation

Standard installation is implemented from a completely writeable (as the user root)

directory tree.

1. Log onto the target computer as root and copy the rpm packages to a tempo-

rary directory.

2. Install the required package with

rpm –ihv --nodeps myAMC.FA_AppAgent-<X.Y-Z>.i386.rpm

After all the required packages have been installed, the start scripts may need to be

copied to the root images of the various node types (Application / Control).

The FA CtrlAgent and FA FrameAgent may run on the Control Node, and only the FA

AppAgent may run on the Application Nodes.

The rpm-installation will be only used for explicit change of single packages. Thereby no

automatic migration steps will be done.

For installation or patch of an existing myAMC.FA Suite in a FlexFrame solution only

EzSetup will be used (see following chapters).

First Steps


2.2.2.4 EzSetup-Installation

2.2.2.4.1 General

The installation or patch of an existing myAMC.FA Suite in a Flexframe solution will be

supported with a backup-script as of FlexFrame V5.2A00-018.

The backup script allows the backup of config and dynamic data from myAMC.FA Mes-

senger stored in database.

The data of Accounting (myAMC.FA_DomainManager) can also be backed up with the

backup script if the component is active in the FlexFrame.

Also the config files of the myAMC.FA_Agents can be saved.

With parameters of the backup script can be selected which data will be saved.

The backup functionality can be used before any config changes to save the actual con-

figuration. But also before every change of the FlexFrame during an update or patch.

The config and dynamic data of myAMC.FA_Messengers and the optional my-

AMC.FA_DomainManager will be exported as XML datafiles.

Before export of the dynamic accounting data from myAMC.FA_DomainManager the size

of the data must be respected.

With the paramater ‚-t‘ the estimated time for the export will be shown.

Bevor using the backup script it must be checked if there is enough space left at storage

for the exported data.

The fa_ezsetup has a new integration with the FlexFrame Installer environment as of

FlexFrame V5.2A00-018.

The new installation mode is described in the FlexFrame manual “Installation of a Flex-

Frame Environment”.

The fa_ezsetup tool supports now the following installation processes.

new FlexFrame installations as of FlexFrame 5.2A00-018

upgrade FlexFrame from FlexFrame < 5.2A to FlexFrame ≥ V5.2A00-018

patch FlexFrame V5.2A to FlexFrame V5.2A00-n with n ≥ 018

The FlexFrame preinstaller and the FlexFrame installer together with fa_ezbackup and

the fa_ezsetup automate the database migration process from mySQL to postgreSQL.

Attention: A patch installation from FlexFrame V5.2A to FlexFrame V5.2A00-n with n ≥

018 needs a manually triggered fa_ezbackup before you can start fa_ezsetup.

Check the fa_ezbackup logfile before you start the fa_ezsetup!

First Steps


2.2.2.4.2 Backup with EzBackup

With EzBackup the config and dynamic data of an existing database, if it is running inside

of FlexFrame, will be saved.

The data will be stored in the following directory:

/opt/myAMC/Backup/[Komponente][VersionString]

Every component will be named with version number and stored in an own subdirectory.

Usage:

./fa_ezbackup.sh -h

Usage: ./fa_ezbackup.sh [ -x ] [ -a | -c | -d ] [ -m | -u ] [ -p |

-y ] [ -h ] [ -o dirname ] [ -l logfile ]

-x execute export

-a export agent config only

-c export database config only

-d export dynamic data only

-m use messenger database

-u use accounting database

-p use postgreSQL

-y use mySQL

-h display this help

-o dirname specify output directory

-l logfile write all logs into this file

First Steps


Example sequence of backup:

1. Change to directory containing the update script

cn1: # cd /media/dvd/FA_Suite/myAMC_FA

2. Run a test without permanent changes:

This option determines which data is involved and the estimated time for the export.

cn1: # ./fa_ezbackup.sh -l /tmp/myAMC_FA.log

If this test did not report errors continue with step #3.

If this test failed you can't continue since the upgrade will fail. Please fix the problem

by help of the mentioned README and start again with step #2.

3. Run the backup.

cn1: # ./fa_ezbackup.sh -l /tmp/myAMC_FA.log -x

4. Check the results.

Check the data in backup-directory: /opt/myAMC/backup

2.2.2.4.3 Standard-Installation with EzSetup

A detailed description of the installation with EzSetup und possible changes and hints can

be found in the readme-file “fa_ezsetup.README.txt”.

1. Change to directory containing the update script:

cn1: # cd /media/dvd/FA_Suite/myAMC_FA

1.1. Make a backup: (Only needed for FlexFrame versions < 5.2A00-018 for database-

migration to PostgreSQL)

Check for errors and estimated time:

cn1: # ./fa_ezbackup.sh

Do the backup:

First Steps


cn1: # ./fa_ezbackup.sh -x

2. Clear the state:

cn1: # ./fa_ezsetup.sh -L /tmp/myAMC_FA.log -S

/tmp/myAMC_FA.state clear

3. Run a test using the -n option of fa_ezsetup.sh:

This option prevents any permanent changes, but simply shows

what would be done.

For destination FlexFrame-version less or equal 5.1:


/tmp/myAMC_FA.state –o 51 -n -v -A install

For destination FlexFrame-version greater then 5.1:


/tmp/myAMC_FA.state -n -v -A install

If this test did not report errors continue with step #4.

If this test failed you can't continue since the upgrade will fail.

Please fix the problem by help of the mentioned README and start again

with step #3.

4. Run the actual upgrade:


/tmp/myAMC_FA.state clear

For destination FlexFrame-version less or equal 5.1:


/tmp/myAMC_FA.state –o 51 -v -A activate –r

For destination FlexFrame-version greater then 5.1:


/tmp/myAMC_FA.state -v -A activate –r

5. Check the results:

First Steps



/tmp/myAMC_FA.state -v status

Statement regarding destination FlexFrame-version (see option '-o' of fa_ezsetup.sh):

Destination FlexFrame-version less or equal 5.1

The file myAMC_FA_Rules.xml in the pool-config will be not overwritten.

The myAMC.FA_DomainManager will be not migrated.

The myAMC.FA_LogAgent will be not migrated.

The myAMC.FA_Suite will be not installed.

Destination FlexFrame-version greater then 5.1

The file myAMC_FA_Rules.xml in the pool-config will be overwritten with the file

from template.

Possibly hold customer specific rules should be saved before and must be taken

over in the customer specific rule-file myAMC_FA_Rules_Custom.xml manually.

Hint for configfiles:

During installation with EzSetup the poolspecifig config-files will be back up, so customer

specific configuration was saved.

The backup of the old version can be found as follows:

/opt/myAMC/vFF/vFF_<POOL>/Migration.<NEWVERSION>.<DATE>/Backup.<OLDVERSION>/config

The migrated files of the new version can be found as follows:

/opt/myAMC/vFF/vFF_<POOL>/Migration.<NEWVERSION>.<DATE>/config

Example:

/opt/myAMC/vFF/vFF_p1/Migration.V90A00K27.2012-08-23/MIGRATION-INSTRUCTIONS.txt

/opt/myAMC/vFF/vFF_p1/Migration.V90A00K27.2012-08-23/Backup.V90A00K26

/opt/myAMC/vFF/vFF_p1/Migration.V90A00K27.2012-08-23/Backup.V90A00K26/.info

/opt/myAMC/vFF/vFF_p1/Migration.V90A00K27.2012-08-23/Backup.V90A00K26/config

TrapTargets.xml

config-specs.xml

myAMC_FA.xml

myAMC_FA_GUI.xml

myAMC_FA_Groups.xml

myAMC_FA_Rules.xml

myAMC_FA_SD_Sec.xml

First Steps


/opt/myAMC/vFF/vFF_p1/Migration.V90A00K27.2012-08-23/.info

/opt/myAMC/vFF/vFF_p1/Migration.V90A00K27.2012-08-23/config

TrapTargets.xml

config-specs.xml

myAMC_FA.xml

myAMC_FA_GUI.xml

myAMC_FA_Groups.xml

myAMC_FA_Rules.xml

myAMC_FA_SD_Sec.xml

2.2.2.4.4 Important for Installation with fa_ezsetup

The installation/upgrade using fa_ezsetup must be performed on the active control node.

2.2.2.4.4.1 For FlexFrame < V 5.2A00-018

For FlexFrame < V5.2A00-018 within an update or patch installation of FlexFrame the

migration of the used database will not be done automatically.

The database creation / updates must be performed manually before using fa_ezsetup.

Necessary databases:

Necessary creation and updates of myAMC.FA_Messenger and my-

AMC.FA_DomainManager database won't be done automatically.

The database creation / updates must be performed before using fa_ezsetup.

This must be done in maintenance mode of an HA-Software.

The myAMC.Messenger / myAMC.FA_DomainManager must be stopped.

Creation of database:

Syntax for mySQL:

mysql -u [user] -p[passwort] < SQL-script.sql

Details:

First Steps


For myAMC.Messenger database:

cd /media/cdrom/FA_Suite/myAMC_FA/Messenger_Scripts/

mysql -u [user] -p[passwort] < CreateAmcMsgmySQL_FF.9.0-03.sql

Always use the newest version of script (example above ‘9.0-03’).

For myAMC.DomainManager database:

cd /opt/myAMC/FA_DomainManager/sql /

mysql -u [user] -p[passwort] < create-amc-MySQL.sql

Update of database:

Syntax for mySQL:

mysql -u [user] -p[passwort] < UpdateAmcMsgmySQL_FF.6.0A12-KXXTo6.0A12-KYY.sql

or

mysql -u [user] -p[passwort] < UpdateAmcMsgmySQL_FF.9.0-XXTo9.0-YY.sql

The Values for ‘XX’ and ‘YY’ must be detected in customer installation.

Value ‘YY’ is from the newest database update script.

For myAMC.Messenger:

cd /media/cdrom/FA_Suite/myAMC_FA/Messenger_Scripts/

For myAMC.DomainManager:

cd /opt/myAMC/FA_DomainManager/sql

Value ‘XX’ is from the installed database version value ‘DBVERSION’.

It can be detected from database like the following example value ‘DBVERSION’:

For myAMC.Messenger: /usr/bin/mysql --user=root --password=passwort --

database=messenger --execute="select * from CNF_CONFIGGLOBAL where

APPLICATION like 0;"

+-------------+-----------------+---------+

| APPLICATION | TYPES | STRINGS |

First Steps


+-------------+-----------------+---------+

| 0 | DBVERSION | 9.0-03 |

| 0 | DBCONTENT | 9.0-03 |

| 0 | POLLEVTMANCYCLE | 5 |

+-------------+-----------------+---------+

Or:

/usr/bin/mysql --user=root --password=passwort --

database=messenger --execute="select STRINGS from CNF_CONFIGGLOBAL

where TYPES='DBVERSION';"

+---------+

| STRINGS |

+---------+

| 9.0-03 |

+---------+

For myAMC.DomainManager: /usr/bin/mysql --user=root --password=passwort --database=amc --

execute="select * from CNF_CONFIGGLOBAL where APPLICATION like 0;"

+-------------+-----------+---------+

| APPLICATION | TYPES | STRINGS |

+-------------+-----------+---------+

| 0 | DBVERSION | 9.0-15 |

| 0 | DBCONTENT | 9.0-15 |

+-------------+-----------+---------+

Or:

/usr/bin/mysql --user=root --password=passwort --database=amc --

execute="select STRINGS from CNF_CONFIGGLOBAL where

TYPES='DBVERSION';"

+---------+

| STRINGS |

+---------+

| 9.0-15 |

+---------+

All scripts between ‘XX’ and ‘YY’ must be executed.

Example for update:

/opt/myAMC/Messenger/Messenger/bin/myAMC.MessengerSrv stop

mysql -u [user] -p[passwort] < UpdateAmcMsgmySQL_FF.6.0A12-K02To6.0A12-K03.sql


…


mysql -u [user] -p[passwort] < UpdateAmcMsgmySQL_FF.6.0A12-K07To8.0-00.sql

mysql -u [user] -p[passwort] < UpdateAmcMsgmySQL_FF.8.0-00To8.0-01.sql


…


First Steps



…


/opt/myAMC/Messenger/Messenger/bin/myAMC.MessengerSrv start

2.2.2.4.4.2 For FlexFrame ≥ 5.2A00-018

With FlexFrame ≥ 5.2A00-018 within an update or patch installation of FlexFrame there

was done a migration of the used database.

This will be done as follows:

With EzBackup the config and dynamic data of an existing database, if it is running inside

of FlexFrame, will be saved.

The data will be stored in the following directory:

/opt/myAMC/Backup/[Komponente][VersionString]

Every component will be named with version number and stored in an own subdirectory.

EzSetup as of FlexFrame V5.2A00-018 has an additional functionality for creation and

initialization of a PostgreSQL database server and to import the data saved with EzBack-

up.

The data for import will be read from the backup directory (last version) and/or from a

template directory. In the template directory as example the trap-rules of my-

AMC.FA_Messenger will be stored as xml file.

2.2.2.4.5 Usage

./fa_ezsetup.sh -h

usage: fa_ezsetup.sh <general options> <command> <command specific args>

fa_ezsetup.sh 1.104

general options:

-h (show this help)

-q (quiet mode)

-v (verbose mode, may be specified multiple times)

-L <file> use specified logfile (implies -l).

If <file> is a directory, the default file name will be appended.

-l log to file and console

-n do not perform permanent changes

-A automatically perform required steps

-R <dir> FA root dir (default: /opt/myAMC)

First Steps


-D <dir> release dir

-P <properties> release property file

-S <statefile> state file

-s do not save state

-C clear state

-o <version> specifies the version. For version 5.1 or older <version>

have to be set to '51'. Otherwise installation fails. (option not given =

default = newest version)

-f force initialisation of databases, discard existing data (messenger

and accounting) -i force installation of packages, even if same version is already

installed

commands:

clear clear installation state

info show info on current environment

status show installation status

check check pre requisites

install install packages to be updated

migrate migrate current settings to new version

autoconfig automatic configuration (if possible)

activate activate new version(s)

version show version of fa_ezsetup.sh

run run commands on application or control nodes

startapps start all myAMC components

This will start all myAMC applications on the control

node

as well as all agents for all pools on the application

nodes

stopapps stop all myAMC components

This will stop all myAMC applications on the control node


nodes

restartapps restart all myAMC components

This will restart all myAMC applications on the control

node


nodes

run fa_ezsetup.sh help <command> for command specific help

First Steps


2.2.3 Configuration

The FlexFrame Autonomy Agents do in most cases not require any additional configura-

tion for use in productive operation.

The myAMC_FA.xml file is stored when installation takes place. This file already contains

a complete parameter set for the operation of the FA AppAgents and FA CtrlAgents. The

services to be monitored and the reaction scenarios which run in the event of problems

are parameterized in this file. The parameters and their default values are described in

section 7.5. The mode in which the agents are to operate is also configured here.

In the course of the startup, in particular the start times and stop times, the function of the

MonitorAlerts, and the times for a reboot and switchover need to be checked. The Mon-

itorAlerts are a component part of the FlexFrame basic installation.

The MonitorAlertTime must always be at least three times as great as the parameter-

ized CheckCycleTime.

In the startup scenarios, the real start, stop, restart and reboot times must be determined

individually for each service type. If the times specified for start, restart, reboot or switch-

over are not sufficient, this can result in unwanted reaction escalations.

Changes in the parameter file become effective only after the agents have been restart-

ed.

The FA migration tool enables a configuration file of an existing installation to transfer the

data automatically to a new configuration file. Parameters which, for example, were not

present in an older version of the configuration file are then initially automatically set to

their default values.

First Steps


2.2.4 Starting and Stopping

During installation, links to the FA Agents’ start/stop scripts were set in /etc/init.d/.

Run this script without any options so that all available options are displayed, e.g. start or

stop.

Example: Starting the FA AppAgent:

/etc/init.d/myAMC.FA_AppAgent start

/opt/myAMC/FA_AppAgent/myAMC.FA_AppAgent start

Example: Starting the FA CtrlAgent:

/etc/init.d/myAMC.FA_CtrlAgent start

/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent start

Example: Starting the FA FrameAgent:

/etc/init.d/myAMC.FA_FrameAgent start

/opt/local/myAMC/FA_FrameAgent/myAMC.FA_FrameAgent start

First Steps


2.2.5 FA-Agents in different FlexFrame Versions

Several changes are necessary depending on which FlexFrame version is used with the

FA-Agents. These changes must be done manually and for each pool. After the configu-

ration changes, the FA-Agents must be restarted.

This can be done with the tool fa_ezmigrate.sh automatically.

It is also possible to do this manually, but it is not recomended.

2.2.5.1 Migration with fa_ezmigrate (recommended)

The migration of the configuration comprises two steps.

Both are supported by the fa_ezmigrate.sh script.

1. Adjust the FA Agent configuration to the current FlexFrame version

cn1: # /opt/myAMC/scripts/fa_ezmigrate.sh srvscripts

2. Reset the FA-Agent Rules to the installed FA Agent version

cn1: # /opt/myAMC/scripts/fa_ezmigrate.sh resetrules

If there are customer specific rules, they must be re-inserted manually after resetting the

rules.

The above commands will execute the migration for every pool and use the FlexFrame

version from /etc/FlexFrame-release.

If you want to migrate not all pools, or you want to migrate to a given FlexFrame version,

you can specify the version or/and the poolname(s) as argument.

For details see the usage of the script (fa_ezmigrate.sh -h):

Usage:

fa_ezmigrate.sh srvscripts [-45n] [<pool> ...]

fa_ezmigrate.sh resetrules [-n] [<pool> ...]

First Steps


arguments for fa_ezmigrate.sh srvscripts:

-4: migrate service scripts for use in FlexFrame version <= 4.2B

-5: migrate service scripts for use in FlexFrame version >= 5.0

default: determine automatically

general arguments

-n process files but don't activate them

-h show usage

-? show usage

if no pools are specified, changes will be applied to all pools.

Description:

"srvscripts" adjusts myAMC_FA.xml to version 4.x (parameter -4) or version 5.x (param-

eter -5).

If the FlexFrame version is not given as argument, it will be read from

/etc/FlexFrame-release.

The original file is kept as backup with the timestamp of the migration.

"resetrules" overwrites the file myAMC_FA_Rules.xml with the version from the configu-

ration template of the propper FA Agent version. The original file is kept as backup with

the timestamp of the migration.

The script returns 0, if all pools could be successfully converted.

The parameter "-n" should be used in both commands, if the file should be migrated or

copied, but not be activated. In this case the file will be written as "my-

AMC_FA(_Rules).xml.temp.<timestamp>". Otherwise the file "my-

AMC_FA(_Rules).xml.<timestamp>" will be used as backup.

2.2.5.2 Installation and migration for central myAMC.FA components

With EzSetup also the central components of myAMC-Suite could be installed, migrate

and activated.

First Steps


• myAMC.FA_DomainManager

• myAMC.FA_Messenger

• myAMC.FA_LogAgent

With the migration the configfiles of the central component will be migrated.

The component myAMC.FA DomainManager and myAMC.FA Messenger use beside

config files also a database to store data. The databases will be also migrated.

First Steps


2.2.5.3 Manual Migration (only for documentation)

2.2.5.3.1 Additional check of files

FA-Agents V9.0 in FlexFrame Version < 5.0A:

The additional check of files for the file ‘/opt/myAMC/scripts/sap/ff_service.sh’ must be

removed, because this file does not exist in a FF < 5.0A.

FA-Agents V9.0 in FlexFrame Version >= 5.0A:

The additional check of files for the files ‘sap*’ must be removed, because these files do

not exist in a FF >= 5.0A. There the new FF-Scripts are used.

The additional check of files for the file ‘/opt/myAMC/scripts/sap/ff_service.sh’ must be

added, because this file is new in FF >= 5.0A.

This can be done with FA-WebGUI (Pool-Configuration: “Additional_Checks”

“File_check”) or manually in file “myAMC_FA.xml“ (see “File_check”).

Example can be found in template:

See file “/opt/myAMC/vFF/Common/.vFF_template/config/myAMC_FA.xml”.

2.2.5.3.2 FF-SAP Start-/Stop-Scripts

New generic scripts for starting and stopping of the SAP-services are available with the

FlexFrame version V5.0A.

The new SAP start- and stopp-scripts was available after installation of FlexFrame 5.0A.

The new FF-SAP-Start-/Stop-interface in the FA-Agents is active after installation of FA-

Agents >= V90A00K07.

Depending on which FlexFrame version is used with the FA-Agents the configuration of

this FF-SAP-Start-/Stop-interface must be changed.

FA-Agents V9.0 in FlexFrame Version < 5.0A:

In this case the following entry must be active:

${ScriptPath}/${ScriptPrefix}${monparam}${ScriptPostfix}

${BYNIDENT} ${SIDENT} ${ACTION}

In this case the following entry must be active for DB2:

First Steps


${ScriptPath}/${ScriptPrefix}db${ScriptPostfix} ${BYNIDENT}

${SIDENT} ${ACTION}

FA-Agents V9.0 in FlexFrame Version >= 5.0A:

In this case the following entry must be active:

${ScriptPath}/ff_service.sh -s ${SIDENT} -t ${monparam}

${BYKEYNIDENT} -a ${ACTION}

In this case the following entry must be active for DB2:

${ScriptPath}/ff_service.sh -s ${SIDENT} -t db ${BYKEYNIDENT} -a

${ACTION}

This can be done with FA-WebGUI (Pool-Configuration: „Autonomy“ „Service Par-

amaters“ „ReactionScript“) or manually in file “myAMC_FA.xml“ (see “Ser-

vice_ReactionScript”).

Example can be found in template:

See file “/opt/myAMC/vFF/Common/.vFF_template/config/myAMC_FA.xml”.

This entry is service-type specific, but there is a default value used if no specific value is

available.

Example from XML-File:

<configentry name="Service_ReactionScript">





<value type="String">${ScriptPath}/ff_service.sh -s ${SIDENT} -t ${monparam}

${BYKEYNIDENT} -a ${ACTION}</value>

</configentry>

2.2.5.3.3 Configuration file for generic services

In case of the customer has done no changes in generic services:

The configuration file for generic services “myAMC_FA_Rules.xml” must be copied

manually from the template in the pool configuration.

First Steps


Example:

cp /opt/myAMC/vFF/Common/.vFF_template/config/myAMC_FA_Rules.xml

/opt/myAMC/vFF/vFF_<POOL>/config

In case of the customer has done changes in generic services:

The configuration file for generic services “myAMC_FA_Rules.xml” must be migrated to

the file in the template.

Save the configuration of generic services.

Copy the file from the template to the file in pool-configuration.

Implement the changes from customer to the configfile of generic services.

Example:

cp /opt/myAMC/vFF/vFF_<POOL>config/myAMC_FA_Rules.xml

/opt/myAMC/vFF/vFF_<POOL>/config/myAMC_FA_Rules_CUSTOM.xml



vi /opt/myAMC/vFF/vFF_<POOL>config/myAMC_FA_Rules.xml

First Steps


2.2.6 Configuration of generic

Generic services can be configured in two different files:

myAMC_FA_Rules.xml: Generic rules general available as best practice which

is central delivered from FlexFrame.

myAMC_FA_Rules_Custom.xml: Customer specific rules.

Therefore the customer specific rules, if used in myAMC_FA.xml, once must be migrated

in an present installation to the custom-rules.

Attention:

The whole pool specific ‚myAMC_FA_Rules.xml‘-files should be saved before call of

EzSetup, if customer specific rules are used.

During migration the ‚myAMC._FA_Rules.xml‘-file will be taken automatically from tem-

plate.

EzSetup assume that customer specific generic rules were transfered to custom-rules-file

before calling.

Case 1: Customer has no changes in generic service configuration:

The configuration of the generic services ‘myAMC_FA_Rules.xml’ must be cop-

ied manually from template in the pool configuration.

Hint: Within installation with EzSetup this will be done automatically.

Example:



Case 2: Customer has changes in generic service configuration:

The configuration of generic services must be manually migrated:

Save the config of the (old) genreric services (old-‚myAMC_FA_Rules.xml‘).

First Steps


Copy the file from template in the pool-config (new-‚myAMC_FA_Rules.xml‘).

Add / migrate the customer specific changes to the customer specific generic

trules in pool-config (‚myAMC_FA_Rules_Custom.xml‘).

Hint: Also within installation with EzSetup this will must be done manually.

Example:

cp /opt/myAMC/vFF/vFF_<POOL>config/myAMC_FA_Rules.xml

/opt/myAMC/vFF/vFF_<POOL>/config/myAMC_FA_Rules_OLD_with_CustomChanges.xml



vi /opt/myAMC/vFF/vFF_<POOL>config/myAMC_FA_Rules_Custom.xml

First Steps


2.3 FA WebInterface

2.3.1 Function

The FA WebInterface visualizes all nodes and services which exist in a FlexFrame sys-

tem insofar as these are monitored by an FA AppAgent. The status, availability and mes-

sages of the FA AppAgents and Fa CtrlAgents are displayed.

2.3.2 Installation

The installation package is called myAMC.FA_WebGui-<X.Y-Z>.i386.rpm.

A prerequisite here is that an Apache Tomcat Servlet Container is installed. Currently

Tomcat >= 6.0.x is supported.

2.3.3 Configuration

Further settings can be made in the files /opt/myAMC/config/FA_WebGui.conf

(general settings, paths, cycle tymes, database settings) and /opt/myAMC/config/amc-users.xml (user administration).

Changes require the FA WebInterface to be restarted or reloaded (e.g. via the Tomcat

Service Manager) or Tomcat to be restarted or reloaded.

2.3.4 Starting and Stopping

The WebInterface can always be reached if the Apache Tomcat is running. This can

generally be started using the script /etc/init.d/tomcat6 start.

The WebInterface can then be reached at the following address:

http://<contol_node>:8080/FAwebgui/

The specified port can be changed in the Tomcat configuration file server.xml.

Prerequisites here are Mozilla >= 1.4.1 or Internet Explorer >= 6.0 and the latest Java

plugin / JRE for Oracle or IBM.

2.3.5 myAMC.FA WebGui and database

The web interfaces use a connection to a database for displaying the message and per-

formance data.

First Steps


myAMC.FA Messenger and the myAMC.FA Accounting Option (my-

AMC.FA_DomainManager) use an own database in a database server. The my-

AMC.FA_DomainManager database can be run optionally at a system outside of the

FlexFrame, for not influencing the productive FlexFrame.

Cardinally the myAMC.FA_WebGui can work with different types of database. This de-

pends on the selected database in the connection property.

In FlexFrame version < V5.2A00-018 normally a mySQL database is used.

In FlexFrame ≥ 5.2A00-018 a PostgreSQL database will be used as default.

2.3.5.1 Installation and configuration of a mySQL connection

The database connection will be configured in the following file:

/opt/myAMC/config/connector.properties

Example:

#connector

#Tue Jan 29 16:16:52 CET 2013

jdbc.url.Accounting_Database=jdbc\:mysql\://localhost\:3306/amc?useGmtMillisForDatetimes\=true

connector.type.Accounting_Database={jdbc}jdbc

jdbc.url.Messenger_Database=jdbc\:mysql\:// localhost \:3306/messenger

connector.priority.Accounting_Database=10

jdbc.user.Messenger_Database=myAMC

connector.type.Messenger_Database={jdbc}jdbc

connector.active.Accounting_Database=true

jdbc.password.Accounting_Database=FlexFrame

connector.name.Accounting_Database=Accounting Database

connector.name.Messenger_Database=Messenger Database

jdbc.driver.Messenger_Database=com.mysql.jdbc.Driver

jdbc.driver.Accounting_Database=com.mysql.jdbc.Driver

connector.active.Messenger_Database=true

jdbc.user.Accounting_Database=myAMC

jdbc.password.Messenger_Database=FlexFrame

First Steps


2.3.5.2 Installation and configuration of a PostgreSQL connection


/opt/myAMC/config/connector.properties

Example:

#connector

#Thu Jan 31 09:48:31 CET 2013

jdbc.url.Accounting_Database=jdbc\:postgresql\://localhost\:5432/amc

connector.type.Accounting_Database={jdbc}jdbc

jdbc.url.Messenger_Database=jdbc\:postgresql\:// localhost \:5432/messenger

connector.priority.Accounting_Database=10

jdbc.user.Messenger_Database=myAMC

connector.type.Messenger_Database={jdbc}jdbc

connector.active.Accounting_Database=true

jdbc.password.Accounting_Database=FlexFrame

connector.name.Accounting_Database=Accounting Database

connector.name.Messenger_Database=Messenger Database

jdbc.driver.Messenger_Database=org.postgresql.Driver

jdbc.driver.Accounting_Database=org.postgresql.Driver

connector.active.Messenger_Database=true

jdbc.user.Accounting_Database=myAMC

jdbc.password.Messenger_Database=FlexFrame

First Steps


2.4 DomainManager

The DomainManager is installed on the Control Node.

The accounting and performance data collected by the FA AppAgents is automatically

adopted by the ITDW and can be visualized and evaluated with the help of the FA

WebGUI with the Accounting and Performance management plugin.

The DomainManager is configured via the file /opt/myAMC/DomainManager/config/

DomainManager.xml. Pool-specific configuration is also possible. Changes to parame-

ters in the DomainManager configuration are dynamically recognized and adopted.

Alternatively to processing through the DomainManager, the files can also be accessed

by an external DomainManager which runs outside of FlexFrame. In addition to this,

extension of the Tomcat server by means of the myAMC.Fileretriever module is

possible. This is optional and not part of the standard delivery.

2.4.1 Database connection

The DomainManager uses a connection to a database to store the performance data.



The myAMC.FA_DomainManager and its database can be run integrated in FlexFrame

or optionally at a system outside of the FlexFrame, for not influencing the productive

FlexFrame.

First Steps


2.4.1.1 Integrated operation of FlexFrame Accounting option

2.4.1.1.1 mySQL database server connection


/root/.odbc.ini

or/and

/etc/odbc.ini

Example:

[AMC_PERF]

Driver = /usr/lib/libmyodbc3.so

SOCKET = /opt/myAMC/mySQLDBs/mysql.sock

Server = localhost

User =

Password =

Database = amc

Port = 3306

2.4.1.2 PostgreSQL database server connection


/root/.odbc.ini

or/and

/etc/odbc.ini

Example:

[AMC_PERF]

Driver = /usr/lib/unixODBC/libodbcpsql.so

Database = amc

Servername = localhost

Username = postgres

Password =

Port = 5432

#ksqo = No

ReadOnly = No

RowVersioning = No

ShowSystemTables = No

ShowOidColumn = No

FakeOidIndex = No

ConnSettings =

Debug = 0

CommLog = 0

First Steps


2.5 Messenger

The Messenger will be installed at control node.

For myAMC.FA_Messenger there is an own documentation. Here only the database

connection will be documented.

2.5.1 Database connection

The Messenger uses a connection to a database to store the message data.



2.5.1.1 mySQL database server connection


/root/.odbc.ini

or/and

/etc/odbc.ini

Example:

[AMCMSG_V10]

Driver = /usr/lib/libmyodbc3.so

SOCKET = /opt/myAMC/mySQLDBs/mysql.sock

Server = localhost

User =

Password =

Database = messenger

Port = 3306

2.5.1.2 PostgreSQL database server connection


/root/.odbc.ini

or/and

/etc/odbc.ini

Example:

[AMCMSG_V10]

Driver = /usr/lib/unixODBC/libodbcpsql.so

Database = messenger

Servername = localhost

Username = postgres

Password =

Port = 5432

First Steps


#ksqo = No

ReadOnly = No

RowVersioning = No

ShowSystemTables = No

ShowOidColumn = No

FakeOidIndex = No

ConnSettings =

Debug = 0

CommLog = 0

First Steps


2.6 myAMC.FA version management

2.6.1 Lifecycle process

The myAMC.FA Suite disposes of an version management component. The version

management component alleviate the lifecycle process of an myAMC.FA installation in

FlexFrame and delivers information about the installed and active versions of the individ-

ual components of the myAMC suite.

The version management was bundled and installed in the rpm myAMC.FA-Suite.

This rpm represent the build version oft the installed myAMC.FA-Suite. So with linux

commands there can be get the info about the installed version of myAMC.FA.

It contains the program ‚versionmanager‘ as a script and resource files with information

for the version management.

For improvement of the lifecycle management there are the following measures:

myAMC.FA-Suite with build version

Versioning of the myAMC.FA config files

Versioning of the myAMC.FA databases

Move of versions for the central myAMC.FA components

Version management for the myAMC.FA-Suite

Update- and migrations-handling for optional myAMC-FA components

The myAMC.FA-Suite consists of different components:

Components, which are running once in a FlexFrame:

myAMC.FA_FrameAgent

myAMC.FA_WebGui

myAMC.FA_Messenger

myAMC.FA_LogAgent

myAMC.FA_DomainManager

Components, which are running once in a pool:

myAMC.FA_CtrlAgent

Components, which are running on every application node:

First Steps


myAMC.FA_AppAgent

The installation and activation of a new myAMC.FA-Suite is divided in 3 phases, to avoid

downtime and to switch back to an older version easily.

Installation of the rpm’s of a suite

Backup of the der existing config files

Migration of the config files if necassary (EzSetup)

Update of the databases (update-scripts)

2.6.2 Lifecycle management files

With the rpm of myAMC.FA-Suite for the version management the following components

will be installed:

Version-Dependency Datei

Version-Component Datei

Versionmanager

o Output des Versionmanagers

The rpm will install these components in the directory ‚/opt/myAMC/VersionInfo’.

2.6.2.1 Version-Dependency file

In this file the dependency to other myAMC.FA components will be found.

The Version-Dependency file shows tabulary the components version and the dependend

component with version information. The file is a csv-file.

First Steps


Example:

If the version or the dependency of a component has changed, this will be added in this

file.

First Steps


2.6.2.2 Version-Components file

The management of myAMC.FA components which belongs to a myAMC.FA-Suite and

there version numbers will be done in the version-components file.

The file is build tabulary as csv-file and can be imported easily in Excel.

File structure of version-componens file.

MYAMCSUITE version und build number if the myAMC.FA-Suite

MYAMCCOMPONENT name of the associated myAMC.FA component

VERSION_KLEVEL version with K-level

POOL_FRAME cebtral or pool specific myAMC.FA component

PARAM_CONFIG version of he associated config files

BASIC_CONFIG version of he associated config files

RULEFILE version of he associated myAMC_FA_Rules.xml

BINARY version for the database (binary)

DBCFG version for the database (config)

DBSTRUCT version for the database (structure)

DBCONTENT version for the database (content)

FF4S_FROM FlexFrame versionen for the myAMC.FA component (from)

FF4S_TO FlexFrame versionen for the myAMC.FA component (to)

There are many of config files in a myAMC.FA-Suite.

The versioning will be done in groups:

Group of base config files: Config files which have only rare changes.

Group of param config files: Config files, which include parameters for the func-

tionality of myAMC.FA. Example: ‚myAMC_FA.xml‘

First Steps


2.6.2.3 Version manager (from FlexFrame V5.2)

2.6.2.3.1 Mission of version manager

The central mission of the version manager, which simplifies the processing and support

of a FlexFrame landscape, is the dynamic detection of all active myAMC.FA-Suite com-

ponents in the different pools of a FlexFrame landscape. There Info will be written to

standard output, or can be redirected into a file.

The version manager (script) will be installed with the myAMC.FA-Suite rpm.

The version manager shows the active myAMC.FA components, the version of my-

AMC.FA config files und the database version.

It determines if optional components of the myAMC.FA-Suite are active.

The info is shown at standard output and can be redirected to a file.

This report can be send to FlexFrame- or myAMC.FA-Support as info about the custom-

ers system.

Usage:

cn1-a:/opt/myAMC/scripts # ./fa_versionsMgr.sh -?

Usage of fa_versionsMgr.sh:

./fa_versionsMgr.sh -h shows this help

./fa_versionsMgr.sh -? shows this help

./fa_versionsMgr.sh [-s separator] [-f outputfile]

default separator is ;

default outputfile is /opt/myAMC/scripts/fa_versionsCustomer.csv and stdout

cn1-a:/opt/myAMC/scripts #

First Steps


2.6.2.3.2 Output of version manager

The output has the following structure:

FRAMEID Frame-ID

FFVERSION FlexFrame-Release

POOL Poolname

COMPONENT myAMC.FA component

VERSION Version and k-level

CNFPARA Version of parameter files

CNFBASIC Version of basic configuration files

RULES Version of myAMC_FA_Rules.xml resp. Logdefini-

tions.xml

CUSTOM_RULES Customer specific Rules active ?

CUSTOMIZED Customer specific changes ?

VERSIONBIN „VersionBin“ from FA_Messenger.vrs resp.

FA_DomainManager.vrs

VERSIONCFG „VersionCfg“ from FA_Messenger.vrs resp.


VERSIONDBSTRUCT „VersionDbStruct“ from FA_Messenger.vrs resp.


VERSIONDBCONTENT „VersionDbContent“ from FA_Messenger.vrs resp.


CHK_CNFPARA Check of „CNFPARA“

CHK_CNFBASIC Check of „CNFBASIC“

CHK_RULES Check of „RULES“

CHK_VERSIONBIN Check of „VERSIONBIN“

CHK_VERSIONCFG Check of „VERSIONCFG“

CHK_VERSIONDBSTRUCT Check of „VERSIONDBSTRUCT“

CHK_VERSIONDBCONTENT Check of „VERSIONDBCONTENT“

CHK_RPM Check of „VERSION“

CHK_SUMMARY overall summary

First Steps


Example: Output of version manager:

fa_versionsCustomer.csv

First Steps


2.7 Usage of Pool-Master

A FlexFrame as a physical unit of servers, storage and network can be logically divided

into separate pools. Each pool has its own set of configuration data and is thus inde-

pendent of any other pool in a FlexFrame. For each pool a number of parameters needed

to configure a FlexFrame individually for each pool.

With the pool master functionality, the configuration data from a pool that has been con-

figured individually for a flex frame system, are copied to a so-called pool master.

When creating a new pool, the configuration data can be used from the pool master to

generate a new pool.

The usage of Pool-Master allows the automation of steps for myAMC.FA after pool crea-

tion.

Therefore a script can be used to create a Pool-Master from an existing pool or to

write/update a pool configuration from Pool-Master.

The script provides the following commands:

Create Pool-Master (directories/files) from existing Pool

Create/Update Pool (directories/files) from Pool-Master

Delete Pool-Master

Important hint:

The Pool-Master contains config files which may differ from version to version, as well as

the info which version of myAMC.FA_App- and myAMC.FA_Ctrl-Agent will be started.

Therefore it is highly recommended after each update/upgrade of FlexFrame or my-

AMC.FA-Suite to update the Pool-Master from an active pool which was migrated during

update/upgrade process.

Poolspecific configuration should be not passed to the Pool-Master, because they will

force needless queries and error messages in the FA-agents.

Example: „Individual Nodes“, „Management-Blades“, „Filer-Check“, …

First Steps


2.7.1 Create Pool-Master from existing pool

Example: Create Pool-Master from Pool1

In this step the config files of pool ‚Pool1‘ will be copied in the Pool-Master directory

(‚/opt/myAMC/vFF/Common/.vFF_Master‘) as well as the necessary directory structures

will be created.

After that, the pool depending config values will be replaced in Pool-Master:

“TrapTargets” of CN will be replaced with placeholders.

File: TrapTargets.xml

“Managementblades” remain (default) or will be deleted (‘-b’).

File: myAMC_FA.xml

“Individual nodes” will be detected from LDAP (default) or will be deleted (‘-i’).

File: myAMC_FA.xml

“Filer-check” The filer name will be replaced with placeholders.

File: myAMC_FA.xml

2.7.2 Create/Update pool from Pool-Master

Example: Deploy Pool_Master to Pool NEU

vFF

opt/myAMC

config

Create Pool-Master from Pool1

Common/.vFF_template/config

vFF_Pool

1

Config

Files

Config

Files

Config

Files

Common/.vFF_Master/config

First Steps


In this step the config files of Pool-Master will be copied to pool ‚NEU‘in the directory

(‚/opt/myAMC/vFF/ vFF_NEU‘)as well as the necessary directory structures will be creat-

ed.

After that, the pool depending config values will be replaced by the pool specific values:

“TrapTargets” of CN will be detected from LDAP.

File: TrapTargets.xml

“Managementblades” remain (default) or will be deleted (‘-b’).

File: myAMC_FA.xml

“Individual nodes” will be detected from LDAP (default) or will be deleted (‘-i’).

File: myAMC_FA.xml

“Filer-check” The filer name will be detected from LDAP.

File: myAMC_FA.xml

vFF

opt/myAMC

config Create Pool NEU from Pool-Master

Common/.vFF_template/config

vFF_Pool1

Config

Files

Config

Files

Config

Files

Common/.vFF_Master/config

config

vFF_NEU

First Steps


2.7.3 Usage of ‘fa_MasterPoolMgr.sh’

Usage:

fa_MasterPoolMgr.sh -h

usage: fa_MasterPoolMgr.sh <args>

fa_MasterPoolMgr.sh -h

fa_MasterPoolMgr.sh -r

fa_MasterPoolMgr.sh -d toMaster|fromMaster -p <poolname> [-b] [-i] [-m writeTrapTar-

gets|modifyTrapTargets]

Description of arguments:

-h (show this help)

-r remove Pool-Master

-d toMaster|fromMaster specify direction (toMaster, fromMaster)

-p <poolname> specify poolname

[-b] Remove the management blade configuration.

(default = Not remove the management blade configuration).

[-i] Remove the 'individual nodes' (Hypervisor-server) configuration.

(default = Get the 'individual nodes' (Hypervisor-server) configura-

tion from LDAP.

[-m writeTrapTargets|modifyTrapTargets] specify the mode for TrapTarges.xml (default=write)

'modifyTrapTargets' requires correct configuration of trap targets in pool before make a Pool-

Master.

Please ensure, that this script will be started at Ctrl-Node.

2.7.4 Automation of steps during pool creation

The script for Pool-Master management can be used for automation of steps during the

pool creation process.

The script ‚fa_MasterPoolMgr.sh‘ can be called during pool creation from „ff_pool_adm.pl

–op add”. Before calling it should be checked whether a Pool-Master exists. If necessary

the Pool-Master could be created from first pool.

The script ‚fa_MasterPoolMgr.sh‘ can also be used for configuration of several pools of

an existing FlexFrame, for deploying the Pool-Master configuration to the existing pools.

Take in mind, that the script only makes automated adaptions for the pool specific values

described above. Other pool specific things must be changed manually.

First Steps



3 Introduction

3.1 FlexFrame Autonomy Architecture

FlexFrame Autonomy is a powerful component for high-availability operation of systems

with distributed instances. A FlexFrame solution consists of storage, application servers

and redundant Contol Nodes. This product has been implemented for this solution com-

prising storage, servers and connectivity. It enables fast and flexible setup of solutions

which offer autonomous functions to simplify and provide flexibility for operating applica-

tions. The figure below shows an overview of the FlexFrame architecture and the associ-

ated FlexFrame Autonomy components:

The benefit of the FlexFrame Autonomy solution lies in the flexibility for integrating new

nodes and instances without changing the configuration.

Components of FlexFrame Autonomy:

FlexFrame Autonomy Application-Agents (FA_AppAgent)

FlexFrame Autonomy Control-Agents (FA_CtrlAgent)

FlexFrame Autonomy Frame-Agents (FA_FrameAgent)

FlexFrame Autonomy WebGui (FA_WebGui)

FlexFrame Autonomy Messenger (FA_Messenger)

mySAP.com®

Application-

server

Database-

server

Storage

Control Nodes

Application Nodes

FA Control Agents

FA Frame Agent (Option)

FA Messenger

FA Domain Manager (Option)

FA Application Agents

FA Config Files

Introduction


FlexFrame Autonomy DomainManager (FA_DomainManager)

FlexFrame Autonomy LogAgent (FA_LogAgent)

The FlexFrame Autonomy copomponents permit highly available, semi-autonomous

operation of distributed applications. In principle the instances can be distributed to any

number of nodes within a FlexFrame solution. The individual services are monitored via

FlexFrame Autonomy Agents. By default, the FA AppAgents currently support SAP cen-

tral instances and SAP application instances, as well as SAPDB and Oracle database

instances.

The FA Frame-Agents are used for monitoring at the control node. The central monitoring

of filesystems requires the FA FrameAgents which are installed at the control nodes.

Introduction


3.1.1 FlexFrame Autonomy Visualization and Reporting

The myAMC.FA WebGui and the myAMC.FA Viewer are the components for the visuali-

zation of the availability and performance data. The myAMC.FA WebGUI is an applet to

be used in a webbrowser. The myAMC.FA Viewer is a stand-alone application, which

can run on any Linux or Windows PC. The two components are necessary because both

concepts add their own advantages. The applet component can be used on every PC

with connection to the myAMC server component without any installation steps, but ap-

plets have some restrictions. An applet can not use the reporting functionality and can not

connect to multiple frames with one applet.

Starting with version 9.0 of the myAMC.FA Suite, myAMC.FA Viewer supports visualiza-

tion of multiple frames.

The visualization architecture is shown in the following picture.

Admin PC

HTTP

DB CI APP

Accounting and

Performance Collets

myAMC.FA WebGui Applet

myAMC.FA Viewer

FA Agents

FlexFrame

myAMC.FA

Messenger DB

Messenger

JDBC

myAMC.FA

Accounting DB

HTTP

LivelistLivelist

FlexFrame ... n

WebserverWebserver

Reports

Multi Frame Technologie

Introduction


3.1.2 FlexFrame Autonomy virtual server and hypervisor

nodes

Since FlexFrame version V5. 0 virtual servers and hypervisor nodes can be visualized in

the myAMC.FA viewer. The existing web service component running in the Tomcat server

periodically collects data of all virtual servers and their hosts.The information is shown in

the system pool “FlexFrame”.

As of FlexFrame V5.3 or FlexFrame Orchestrator V1.0 ESX or KVM can be used as hy-

pervisor.

On the hypervisor the virtual servers ESX-VM and KVM-VM are running.

The representation and visualization is basically identical for both types of hypervisor.

The corresponding overlay icon is depending on type "ESX" or "KVM".

The myAMC.FA CtrlAgent monitors the virtual servers belonging to its pool. If a virtual

server is not available any more, it is switched offwith the shutdown facility for virtual

servers.

If powering down a virtual server fails, there are several cases:

the hypervisor node runs other VMs without problems

Introduction


the hypervisor node does not host any other VMs

other VMS on the hypervisor node also failed.

An hypervisor node may host virtual servers from multiple pools.

As the myAMC.FA CtrlAgent knows only nodes running in its own pool, bust the hypervi-

sor node possibly also operates virtual nodes for other pools, the permission to turn off

the hypervisor nodes from the central hypervisor nodes monitoring service (part of the

myAMC.WebGui server is part of the Tomcat) is requested. If the permission couls be

obtained, the external switchover can be performed as planned. Can the hypervisor node,

for example because other VMs, not be switched off, the external switchover is aborted.

Introduction


3.2 Pool Creation and Grouping

FlexFrame Autonomy Version 2.0 permits pool creation and grouping functions to be

implemented.

3.2.1 Virtual FlexFrame Autonomy Pools

A pool is the assignment of hardware resources to a virtual FlexFrame Autonomy pool.

From the viewpoint of autonomy and of the high-availability functions, an Autonomy pool

is an independent structure. In a standard installation of Version 1, all resources of a

FlexFrame solution are managed in a single pool. Configuration of the pools takes place

directly with the configuration of FlexFrame in the LDAP. The FA AppAgents ascertain

the pool affiliation at startup. Configuration of the FlexFrame agents always relates to one

pool, i.e. there is one directory tree with the parameters and configuration data for each

pool.

In a pool, the FA Agents provide the autonomous functions restart, reboot and switchover

of services and nodes. These reactions no longer relate to all nodes of a FlexFrame solu-

tion, but only to the set of nodes which belong to the same pool.

Pool creation results in virtual FlexFrame Autonomy pools being created, each of which

performs autonomous functions independently of other pools which exist in the same

FlexFrame solution.

A FlexFrame Autonomy pool always consists of one Control Agent and n Application

Agents. Each Control Agent is responsible only for the Application Nodes which belong to

its pool and shares a joint config and data directory with its Application Nodes. For each

pool it is thus possible to parameterize autonomous behavior which is independent of

other pools.

The flexibility and security of a virtual FlexFrame pool is based on two major features

which the FlexFrame Autonomy Agents provide.

A Control Agent for each virtual FlexFrame Autonomy pool

Each Application Agent is provided with a flexible assignment to the pool and thus to the

Control Agent with which it interworks.

The use and interleaving of these two new options with the FlexFrame basic functionality

offers a large number of new options to enhance the flexibility in server farms.

The virtual FlexFrame Autonomy pools provide the option of simple and secure operation

of multiple Autonomy clusters which run in parallel and simultaneously in a distributed IT

infrastructure.

Introduction


A virtual FlexFrame Autonomy pool offers the advantage of complete separation of all

reactions and the associated parameter sets for the start and stop times. FlexFrame

Autonomy can also be completely disabled for a virtual pool (e.g. for service and

maintenance) without affecting any other virtual FlexFrame Autonomy pool which is

running in parallel.

FlexFrame™

Autonomy Pooling / Grouping

Data

Config Logfiles

vFF Pool1

Introduction


A virtual FlexFrame does not share its FlexFrame Autonomous Agents with any other

virtual FlexFrame. In this way, depending on the configuration, the virtual FlexFrame

Autonomy pools could use different binary statuses.

The pool FlexFrame and Adminpool are special pools.

The Adminpool is a pool for spare nodes which are pool independent..

The pool FlexFrame is a system pool established on the control node to collect infor-

mation for the whole frame.

Introduction


3.2.2 Grouping

FlexFrame offers grouping functions for flexible server farming. Grouping enables nodes

and services within a pool to be assigned to different groups. A group is therefore always

a part of a virtual FlexFrame pool.

The configuration information for the groups is stored in the myAMC_FA_Group.xml file.

The entries in this file are taken them over from the LDAP directory. The configuration is

always done via LDAP by specific assignment. The ability of the generic assignment of

theFA Agent is not supported in FlexFrame.

3.2.2.1 Configuration in the LDAP Directory

The group information are stored in the LDAP directory. When the agents are started, the

group information is read directly from the LDAP directory and be stored in an xml file myAMC_FA_Group.xml.

3.2.3 Pool Types (Pool Independent Spare)

Pools can differ in admin pools and productive pools. The FA Agents have different rules

depending on in which pool type the FA CtrlAgents are acitve.

The admin pool in contradiction to the productive pool has the function to provide spare

nodes for productive pools. If a node fails, the FA Agent requires a spare node from

spare pool. Saving spare nodes and provide them pool overlapping for multiple pools

represents the ability of that functionality.

The property of pool-overlapping spare nodes can be activated/deactivated for each pool.

The pool type is needed for the use and control of admin pools.

Pool type

productive pool (default value)

Adminpool

With a further parameter can be controlled after which time period the spare nodes in the

admin pool start cyclic with the check of availability and performance and stop again

automatically.

Both Adminpool and productive pool in principle use the same versions of FA Agents.

The behavior of the agents depends on the configuration.

3.2.3.1 Spare Node from Adminpool

In the actual version of FlexFrame pools on the level of the FA Agents there is no infor-

mation exchange. By the implementation of pool-overlapping spare nodes they can oper-

Introduction


ate in a collective pool. When a pool needs a spare node, it will be requested from the

spare-pool (Adminpool) and add into the pool/group, which need it.

In a standard FlexFrame installation each pool has one FA CtrlAgent. Therefore this FA

CtrlAgent is only concerned for one pool. There is no interaction with FA CtrlAgents of

other pools.

With the attachment of admin pools, which provides spare for other pools, there is an

information chanel in form of a collective directory. In this directory the FA CtrlAgents can

achieve information of the order of the spare nodes.

Analogical to the so far implemented procedure for reboot and switchover, there will be a

pool-overlapping directory, in which all FA CtrlAgents from each pool can read and write.

In this directory each productive pool gets a switchover directory according the today

actual existing pool specific directory for switchover testaments.

Every pool has its own FA CtrlAgent. This agents work focuses after rules. With the im-

plementation of productive and admin pools as pool type parameters the differing rules of

the FA CtrlAgents become activated dependent on which role the FA CtrlAgent is used.

3.2.3.2 Architecture and Sequence - Pool Independent Spare Node

Procedure spare node from spare pool :

DB

SCS

copy Testament

Testament

SCS

SpareAP

Spare

Spare

Spare

Controlnode

Cont Cont Cont Cont

no local candidate

SparePool eintragen

Sparenode

liste

Inite_sparenode

Delete_sparenode

Change_sparenode

Init_Sparegroup

Introduction


In a productive pool a node fails. The productive pool has no available spare node and no

other node applies for the task. That is proved from the FA CtrlAgent by means of the

configured switchover rules.

The check can have the following results:

The failed node or service has low prior and there is no need to make the node/service

available again.

The services on the failed node have high prior, there is a need for a spare node to make

the services available again. If in the pool/group of the failed noted exist no spare node,

that means there is no applicant, then the FA CtrlAgent decides to require a spare node

from the spare pool.

The FA CtrlAgent in the admin pool has an selection rule to identify the spare node to

transfer to the productive pool. For that the FA CtrlAgent manages a dynamic list with all

activated spare nodes and their selection attributes. A node in the adminpool is activeat-

ed as spare node through a successful boot process in the spare group of the adminpool.

The spare node list is not persistend. FA CtrlAgent initalies every cycle time a boot test of

every node in the spare group of the adminpool. Only nodes which can be booted suc-

cessfully in the MaxReboottime remain in the spare pool list.

The FA WebGui visualizes all nodes nodes in the admin pool, to differentiate activated

spare nodes there is an additional overlay icon in the node tree. Nodes without such a

overlay icon in the tree are not useable as pool-overlapping spare nodes in the admin

pool.

3.2.3.3 Competing Access to Spare Nodes

The FA CtrlAgent of a productive pool transmits all requirements which can not be ac-

complished locally to the spare pool. For the present the FA CtrlAgent does not imple-

ment a further preselection.

As long as no specific perimeter exists all pools have priority 1 through default. Thereby

applies first. With heavier failure scenarios etc. demands for spare nodes will come from

many systems simultaneously. If thus all pools have the same priority then applies the

system or the service priority which already can be configurated today.

3.2.3.4 Providing of Spare Nodes in the Adminpool

The installation and configuration of a spare node in an admin pool for the use as pool-

overlapping spare node is described in the FlexFrame Admininistration and Operation

guide. A spare node in the spare pool is only visible and usable if it has been running at

least once and thus enabled the FA Agents to dynamically determine the output data of

the node. The node will be recorded in a list administered by the FA CtrlAgent during the

initialization.

Introduction


Using a parameter the FA Agents are able to cyclicly check and then stopp again the

availability and the output data of the spare nodes through booting.This guarantees the

actuality of the sparenodelist and the belonging outputdata.

This init- or checkrun can be accomplished for all spare nodes of the spare pools through

a script. Depending on the parameter one or all nodes of the spare group will be initia-

lized.

Introduction


3.3 Service Classes

The service classes are required for the prioritized operation of individual services or

systems. A service is defined as an application instance which must be identified unam-

biguously and which can be started and stopped individually, e.g. central instance, appli-

cation instance or database instance (CI, AP DB).

A service class defines the minimum requirements which must be provided when services

are taken over in the event of a switchover.

When multiple nodes fail simultaneously, the spare nodes in the group take into account

the priorities of the services which have failed. First all services with priority 1 are taken

over, and only then all services with a lower priority. It will be possible to extend the at-

tributes of a service in the future, as already shown in the examples (e.g. operating sys-

tem).

A system is a logical unit consisting of multiple service instances which together define a

system. In an SAP system these comprise the database instance, central instance and

application instances.

The following attributes are defined in the service classes:

service-priority

service-powervalue

In the future it will be possible to enhance such a service class by further attributes which,

for example, define the operating system required by a service or the number of CPUs or

the performance requirement of the service.

3.3.1 Service Priority

The highest service priority is 1. Every service is assigned this priority by default, i.e. if no

service classes are defined, all services have the priority 1. The higher the number, the

lower the priority of a service.

Priority 0 has a special status. Setting priority 0 for a service class enables the autono-

mous functions to be disabled for a service.

The service priority is evaluated for all autonomous reactions. If, for example, a service of

a productive system and a service of a test system are running on the same node and the

test system’s service is assigned priority 5, this service is not executed because the pro-

ductive system’s service which is functioning without error has the higher priority of 1.

3.3.2 Service Power Value

The service power value specifies for a service a performance number which defines the

maximum performance (SAPS) required by this service.

Introduction


This value is used for takeover scenarios; the add rule requires the service power value

A failed service with a performance value of 50 can, for example, also be taken over by a

node which still has at least 50 of its maximum performance number free.

3.3.3 Class Creation Rules

A service belongs either to the default class which always exists or it can be assigned

unambiguously to another class by evaluating the aforementioned variables.

3.3.4 Testament Types

The switchover scenarios use testaments to transport the service information to other

nodes. The creation of the testaments can be node-based or service-based. With node-

based testaments, all services of a node always come together to the takeover node.

With service-based testaments the services could be taken by different nodes.

The parameter for the testament type and the takeover rules therefore strongly influence

the possible takeover scenarios.

3.3.4.1 Service-specific Testaments

Service-specific testaments are used for services which require individual takeover sce-

narios.

3.3.4.1.1 Enqueue Service in Case of Replicated Enqueue

The enqueue service with replicated enqueue service has its own service type. This spe-

cial testament is built dynamically if a replicated enqueue service exists. For this service-

specific testament, the service-based takeover rule applies. Only nodes with a replicated

enqueue service can apply.

Introduction


3.4 FA Configuration, Work and Log Files

The figure below provides an overview of the configuration and log files which are gener-

ated by FA components and stored on the common file system. These files also form the

permanent memory which is required, for example, to restore the services needed when

a system is rebooted.

In a standard FlexFrame installation the following directories exist for myAMC.FA. All the

directories and files required for the myAMC.FA software are located here.

directory structure: /opt/myAMC/

and /opt/local/myAMC/

./FAwebgui FA web interface

./vFF/Common/.vFF_template.<version> Template for pool directories

./vFF/Common/.vFF_Master Pool-Master

./vFF/SpareRequest Requests for spares from sparepool

./vFF/vFF_<pool_name> Pool directory

config Pool-specific configuration data

log Log files

log/AppAgt Log files of Application Agents

log/CtrlAgt Log files of Control Agent

log/FrameAgt Log files of Frame Agent

log.common Common log files

data/FA/ FA data directory

data/FA/blackboard Blackboard directory

data/FA/livelist Live list

data/FA/servicelists Service files of all nodes

data/FA/servicelogs Service files of all nodes (history)

data/FA/xmlrepository XML files for the web interface

data/FA/reboot Reboot files for all nodes

data/FA/switchover Switchover files for all nodes

data/FA/performance Performance and accounting files

A detailed description can be found in chapter 10.3.

Introduction


3.5 Systems

A system is based on several services which belong to a logical group. SAP systems are

an example of logical systems. The services of such a system can be distributed in one

pool on several Application Nodes.

The FA_AppAgents identify the services and the system they belong to autonomously

and they identify standard SAP services automatically.

3.5.1 Service Types

The FA_AppAgents are able to identify standard SAP services and the hierarchy in a

logical SAP system. For these services the FA_AppAgents do the autonomous reactions

restart, reboot and switchover.

DB, CI, APP, J, JC, SCS, ASCS, ERS, LC, MDS, MDIS, MDSS, MDS71, XPR, ...

Version 3.0 of the FA Agents can monitor the service types SCS and ASCS with replicat-

ed enqueue service (ERS). The detection of SCS/ASCS with or without replicated

enqueue service is done automatically.

As of FlexFrame version 5.0A and FA Agent version 9.0, the additional service types

MDS, MDIS, MDSS, MDS71, XPR, ... are supported.

As of FlexFrame 5.3 / FFO 1.0 the additional service types BOBJ 4.1 and SAP HANA are

supported.

3.5.1.1 Replicated Enqueue Service ERS

Version 3.0 of the FA Agents can monitor the service types SCS and ASCS with replicat-

ed enqueue service (ERS) scenarios.

For an SCS or ASCS service there is an ERS on which the enqueue table is replicated.

The detection of SCS/ASCS with or without ERS is done automatically.

Details of the reaction rules are described in chapter 4.

3.5.1.2 Solution Manager Diagnostic (SMD)

Version 9.0 of the FA Agents can monitor the Solution Manager Diagnostic (SMD).

As of FlexFrame V5.2A00-018 the myAMC.FA Agents support host based SMD agents of

version 7.3.

SMD 7.3 agents can work as host based and as service based agents. These require the

existence of a FlexFrame version ≥ 5.2A00-018.

In FlexFrame < V5.2A00-018 the SMD 7.3 agent can only be used with project specific

release and a special configuration. Details of the reaction rules are described in chapter

4.

Introduction


3.5.1.3 Live Cache

With version 3.0 of the FA Agents it is possible to integrate the live cache into the stand-

ard autonomy scenarios. The FA Agents offer the standard autonomy functions restart,

reboot and switchover for the live cache.

A speciality of the live cache is the possibility to stop it from the SAP GUI. For this restart

scenario you have to check the restart times of the live cache, otherwise this scenario

cannot be diagnosed (recognized) as a fault of the live cache.

3.5.1.4 BOBJ 3.2 and BOBJ 4.0

The service BOBJ 4.0 can be detected since FlexFrame V5.1A.

The myAMC.FA Agents therefore has in FlexFrame V5.1 an external rule in file my-

AMC_FA_Rule.xml.

Since FlexFrame V5.2A this rule for BOBJ 4.0 is no more in file, it is an internal rule.

At migration from FlexFrame V5.1A to V5.2A the rule file myAMC_FA_Rule.xml must be

replaced with the rule file for FlexFrame V5.2A.

This will be done automatically during installation with EzSetup.

Important hint:

Also if FA-Agents for FlexFrame 5.2 are used in a FlexFrame 5.1 the my-

AMC_FA_Rules.xml may not include an active BOBJ40 rule.

A potential includes rule must be inactive.

Example for active BOBJ40-rule:

<section type="Service" name="SRV_BOE4">

<description> Created: Feb 2012 </description>

<attribute name="Active">1</attribute>

Example for inactive BOBJ40-rule:

<section type="Service" name="SRV_BOE4">

<description> Created: Feb 2012 </description>

<attribute name="Active">0</attribute>

Introduction


3.5.1.5 SAP HANA

The SAP HANA database server needs a lot of memory and might need a very long time

for getting started.

Therefore, the default start-/stop-/restart- monitoring times were significantly increased.

Service_MaxRestartTime = 3100 Sek

Service_MaxStartTime = 2700 Sek

Service_MaxStopTime = 400 Sek

In case of SwitchOver there are different TakeOver strategies (see chapter 4.4.2) without

hardware check.

In case of "pool independent spare" procedure (see chapter 4.4.4) a similar AN was se-

lected according to certain rules (CPU-arch, HBA, 10GBES, SAPs).

In both cases there is no check of the necessary memory expansion. Thus, it is quite

possible that an AN with too little memory takes over the failed services and therefore

SAP HANA may not start.

Hint:

In case of switch over an AN (SwitchOver or pool independent spare) it is not guaranteed

that a node with similar memory expansion takes over the services of the failed node.

There is also the possibility that an AN takes over with too little memory.

Therefore, the user must ensure that a suitably powerful node takes over the services of

the failed node.

This can be done by appropriate grouping of SAP HANA services, or general use of simi-

lar nodes.

Homogeneous hardware with SAP HANA certified node should be used.

Introduction


3.6 Generic Services

Generic services are services which are not integrated into the FA_AppAgents autonomy

rules. With generic services it is now possible to integrate other virtualized services into

the autonomy monitoring and reaction scenarios.

A generic service is a logic application suite consisting of one or more subservices.

For this purpose a generic service is defined through a set of parameters which are used

for its identification and which generate the service states. The description and definition

of a service is arranged in several models:

Service state model

Service detection model

Service reaction model

3.6.1 Service State Model

The autonomy scenarios are based on an defined state model. The standard service

state model uses the following states:

Starting

Running

Restarting

Stopping

Error

Warning

NoWatch (Attribute of a service)

Reboot

SwitchOver

Some states are detected automatically, but for setting other states there is an external

event neccessary. This will be done due to an eventscript, which will be started during

start of the startscript of SAP service. The eventscript will be visible for a defined time (30

sec) in the process list.

The eventscript is called monitor_alert / monitor_info and will be called with pa-

rameters which indicate the service calling for the external state (start, stopp, restart,

watch, nowatch).

The state changes are initiated through events from an event script or through detection.

For a generic service, implementation and integration in the standard start/stop procedure

of the service is necessary. The standard state model knows the following events:

Start

Stop

Restart

Introduction


Error

Watch

Nowatch (Attribute of a service)

3.6.2 Service Detection Model

The service detection model provides the basis for identifying the service and building the

state model. A service detection model needs the parameters for the identification of the

service components. The parameters are the subservice and the processes of the sub-

service. For this there are parameters for hierarchy and process count. There is also a

process filter and exception rules, to avoid ambiguities.

3.6.3 Service Reaction Model

The service reaction model defines the reaction and the connection to the start, stop and

restart scripts. The reaction API has the parameters Script and Parameter:

Script The call reference for the script

Parameter Set of parameters for the script

The FA-AppAgents reaction API provides a set of parameters, which can be used as call

parameters in service-specific scripts:

@{SIDENT}@ Parameter for the (SID, in upper case)

@{sident}@ Parameter for the (SID, in lower case)

@{SRV}@ Parameter for service name (in upper case)

@{srv}@ Service name

@{SRVDISP}@ Display service name (in upper case)

@{srvdisp}@ Display service name

@{NIDENT:2@ Instance number (two-digits)

Introduction


3.7 FlexFrame Performance and Accounting Option

The FA Agents provide optional performance and accounting data. The agents collect

node-, service- and group-based information.

The FlexFrame performance and accounting option requires the activation of additional

services on the Control Node. This service does a performance and accounting calcula-

tion of the raw data.

The FA Agents produce performance and accounting collections in the data directory of

the pool. There are 3 types of collect data

Collections per node with the name pattern Perf_Node~<node_name>.prf.<number>.col

Collections per service group with the name pattern Perf_Group~<node_name>.prf.<number>.col

Collections per service with the name pattern Perf_Service~<SrvType>~<SID>~<ID>.prf.<number>.col

The number and size of the collections produced by the FA Agents can be adjusted. In

the standard adjustment there are in each case 10 collections per service or node in-

stalled. This results in a ring buffer of data automatically reorganized by the agents. For

the sizing it is possible to calculate the required storage size through the number of

nodes and the size of the report cycle.

The parameters of the DomainManager and of the backup routine have to be configured

in a way that the raw data can be safely processed before being overwritten by the FA

Agents.

Introduction


The following graphic shows the architecture of the performance and accounting option.

3.7.1 Performance Option

The performance option measures several performance values. For all measured values

there is a minimum, average, maximum and total value. This data is supplied in absolute

as well as relative form. The performance option enables monitoring and evaluation of the

server and services over a longer period of time. For every node the following data are

available as a minimal, average and maximum value:

load of SAP-, database- or generic services

other services

Machine idle

By using the generic services, the granularity of the performance values will be

increased. The data of the performance and accounting option can be directly visualized

with the FlexFrame FA Web GUI with performance and accounting management plugin.

The granularity of the view and the timespan can be freely defined.

Service Groups

Services are combined to form groups through specific criteria. This enables the group-

aggregated evaluation of the recorded data.The collected data is aggregated per report

cycle and is created for every node. By default the following groups exist:

mySAP.com®

Application-

server

Database-

server

Network

Storage

DB CI APP

FA-Application

Agents

FA- Performance

and Accounting

Service

ITDW

Accounting and

Performance Collets

Introduction


SAP SAP services

DB Database services

IDLE Share (proportion) of the free CPU capacity

OTHER Sum of all processes not belonging to a defined group

It is possible to define further services and assign them to existing or new groups.

3.7.2 Accounting Option

The accounting option is, like the performance option, an optionally activatable part of the

FA Agents. The production of the accounting data is a multistage process determining

accounting data through aggregation and analysis of the recorded raw data.

The accounting data is determined on the basis of SAPS values. SAPS is the measured

size used for the sizing of a server for the SAP operation. SAPS values are only available

within the scope of a defined benchmark with defined SAP transactions.Therefore only

SAPS equivalents can be produced and calculated during the operation. For this purpose

the agents dynamically evaluate information on the SAP version and hardware-workload

data and use this to calculate the SAPS equivalent values.

Important parameters for the accounting are detection and report cycles. The detection

cycle defines the number of measurements within a report cycle. The minimum, maxi-

mum and average values are calculated on the basis of individual measurements for a

report cycle. The detection cycle therefore always corresponds to the detection cycle of

the FA Agents, which is also parametered for the autonomy function.

The following figure shows the ascertainment and calculation of values with regard to the

detection cycle and report cycle.

Time-

stamp

Host 1Backup

Host 7ASCSP22

Host 3

Host 6

Host 5

Host 4

Host 3

Host 2

Host 1

Hostname

xy

SCSP22

JCP22

JP22

APPP22

CIP22

DBP22

SAPS

%

SAPS

abs

Mem

%

Mem

Kb

CPU

%

CPU

ms

ServiceSystem

SID

Time-

stamp

Host 1Backup

Host 7ASCSP22

Host 3

Host 6

Host 5

Host 4

Host 3

Host 2

Host 1

Hostname

xy

SCSP22

JCP22

JP22

APPP22

CIP22

DBP22

SAPS

%

SAPS

abs

Mem

%

Mem

Kb

CPU

%

CPU

ms

ServiceSystem

SID

Min, Max, Avg, Totalper Report-cycle

Introduction


3.7.2.1 Automatic Calculation of SAPS Values

The SAPS calculation is based on the automatic and dynamically determined workload

ability of a node. Based on a variety of technical features such as cache, CPU, hyper-

threating etc. and the possibilities of the operating system to use these, modern servers

can, come to wrong assumptions concerning the workload abilities of a node. In these

cases the automatic valuation can result in defective workload calculations.

If the internal automatic ascertainment of the SAPS value results in defective values, the

manual SAPS calculation can be used.

3.7.2.2 Manual Calculation of SAPS Values

If the maximum workload number of a server could not be correctly determined via the FA

AppAgent the workload number can be individually defined for each node. The workload

values are then calculated using the prepared workload data. In this way the individual

particularities of the workload abilities of a node can be taken into consideration. For this

purpose, however, the workload values for each node have to be entered manually.

3.7.3 Billing

Using another calculation stage, chargeable workload units can be calculated from the

SAPS-based accounting data. For the calculation, a range of parameters enabling differ-

entiated pricing of the workload used can be set.

In the default mode, all systems and services are charged at the same value every time.

With the help of the FlexFrame Control Center accounting plug-ins, the pricing can be

determined through additional configuration settings.

Detection cycle

Default 10 sec

Max

workload

Min

workload

Report-cycle 1 min

SAPs

t

Total work

Server capacity

Introduction


Therefore the following statements are necessary:

Service contract no.

System ID

ServiceID

Date range

Day type, e.g. weekday, holiday, weekend

Time of day, e.g. daytime, nighttime operation.

The billing table enables very differentiated billing of accounting data as far as the service

contract level. By way of system, service time, time of day and time types, different

service contract items with various workload prices can be used.

CPU/ SAPs Values

0,15€

0,20€

0,30€

0,15€

0,25 €

Accounting

Price

Sapsrule

Sapsrule

Sapsrule

Sapsrule

Sapsrule

Service-

levelrule

SC_12345

SC_12345

SC_12345

SC_12345

SC_12345

Service-

Contract

Standard

Standard

Standard

Standard

Standard

Accounting

rule

24:0000:00daily01.01.210001.01.2006DBP23

24:0000:00daily01.01.210001.01.2006otherP23

24:0000:00daily01.01.210001.01.2006allQ22

P22

P22

System

ID

all

all

Service

typ

00:00

00:00

from

Time

24:00

24:00

to

Time

weekend01.01.210001.01.1900

workday01.01.210001.01.1900

UnitdayTypetoDatefromDate

0,15€

0,20€

0,30€

0,15€

0,25 €

Accounting

Price

Sapsrule

Sapsrule

Sapsrule

Sapsrule

Sapsrule

Service-

levelrule

SC_12345

SC_12345

SC_12345

SC_12345

SC_12345

Service-

Contract

Standard

Standard

Standard

Standard

Standard

Accounting

rule

24:0000:00daily01.01.210001.01.2006DBP23

24:0000:00daily01.01.210001.01.2006otherP23

24:0000:00daily01.01.210001.01.2006allQ22

P22

P22

System

ID

all

all

Service

typ

00:00

00:00

from

Time

24:00

24:00

to

Time

weekend01.01.210001.01.1900

workday01.01.210001.01.1900

UnitdayTypetoDatefromDate

Aggregationcycle

AccountingcycleAccountingreport

Introduction


3.8 Parametricable Value Observation and Service Level Profile

The FA suite offers the possibility of generating the production of events e.g. on trans-

gression of a defined limiting value. Parametricable events are directly evaluated and

dispatched by the FA Agents.In contrary profiles are evaluated by the domain manager

during processing of the performance-, capacity- and accounting-collections, generated

on transgression of adjusted levels of belonging events and dispatched as SNMP trap.

3.8.1 Monitoring of file system and directoy utilization

From version 8.0 on the FA Agents are able to generate parametricable events. On those

terms observable objects and their limiting values can be registerd in a file. This function

exists for the observation of file systems. Obervable file systems as well as the limiting

values for size and increase are registered in a regulation file.On transgression of the

limits a belonging event is generated and dispatched.

3.8.2 Performance Profile, Capacity Profile, Accounting Pro-file

Profiles in which limiting values for the CPU consumption and SAPS consumption can be

defined, are defined for the performance data and accounting data of the FA Agents. On

transgression of the limiting values a SNMP trap is generated.The definition of the pro-

files follows with the standard FA configGui.

Performanceprofile

Capacityprofile

Accountingprofile

The profiles can be defined on service level, system level and node level. The user can

individually decide which profiles exist. A profile can apply to all systems, nodes and

services or pool-, system-, service-, group- or node-specific, depending on the hierachical

level of the profile.

The evaluation of the performance, capacity and accounting profiles ensues through the

domain manager.

The domain manager reads the collections in. The addition and condensation to higher

condensation-based or system-based values follows through the aggregation in the do-

mainmanager.

Introduction


3.9 FlexFrame File System Utilization and File Space Accounting Option

This function is available from version 8.0 of the FA Agents on and enables the observa-

tion of file systems and the node- and system- or service-based accounting of the file

spaces which is needed because of defined files or registers. The fill degree of file sys-

tems on local servers and also on storage systems (NAS and SAN) can be supervised

through the file space accounting system. On reaching a specific percentual threshold

this system will generate a report.

The tasks of the file space management are on the one hand the monotoring which ena-

bles the alerting and reacting in due time if not enough file space is at disposal and on the

other hand the processing of the file system data to the accounting and to the evaluation

for the storage capacity planning.

The following graphic shows the architecture of the file system utilization:

The data for file system utilization and file space accounting can be determined with the

FA AppAgents and the FA FrameAgents.The FA FrameAgents are installed on the

Control Node which allows locally personal parameter files for each. The FA AppAgents

have a pool-specific parametrization meaning all FA Agents of one pool will be using the

same parameter file.

The collecting and processing of the data can be done frame-, node- and system- or

service-specific. The parameter sets and rule sets for the file system utilization allow very

granular definition in which cycles and for which file system or file the storage consump-

tion and the increase will be ascertained.

mySAP.com®

Application-

server

Database-

server

Network

Storage

DB

Service

Node

FA-App Agents

FA- Domain

Manager

ITDW

File utilisation Collections

FA-Frame Agent

Vol FF

Vol 0

System

Group

Frame

Group

Introduction


Statistic definition in the ruleset

The observed file systems, registers and files can be staticly defined in the rulest, i.e. the

start of the FA Agents activates the collection of the data and the calculation for every

report cycle.

Dynamic definition in the ruleset

The ascertainment of the file systems, directories and file data can also be achieved in

dependency to systems and/or specific service types. The generation of the file system

utilization data ensues only if the defined system or a specific service type actually runs

on a node. The FA AppAgents ascertain through the dynamic triggering.

The ressource requirements and time requirements for the ascertainment vary depend-

ing on the size and the number of the monitored file systems, directories and the number

of files. To ensure the use of the functionality in very big SAN-based or NAS-based stor-

age infrastructures a convenient and adapted configuration of the rules and cycles for

collecting and processing of the file system data and file data is necessary. For that the

FA Agents offer the configuration of two different cycles for collecting of the data and a

file system detector which ascertains resource savingly only the consumptions on file

system level and does not execute file-based evaluations.

Another aspect is the visibility of the file systems and directories from the view of one or

more Application Nodes. Normally all nodes of a frame or pool use the same file system.

It is recommended to collect this information with the FA_FrameAgent which will be in-

stalled on the Control Node and which is able to execute this task centrally for a whole

frame.

As soon as a file-based or directory-based supervision or accounting is exspected, the

configurated rules have to be fitted to size and number of the storage systems. Otherwise

a successful implementation and use is not possible.

The directory detector and file detector is a dependent detector, i.e. it requieres infor-

mation of the file systems to supervise the values for individually defined directories or

files.

With that the register detector is able to work from the view of the Application Nodes or

from the view of the Control Nodes. If used on the Application Node generic information

from the services can be used to ascertain the directories, i.e. this function is better suited

for the use of the directory monitored. With that the monitored directories can be generi-

cally derived from the service information. Thereby e.q. data directories and log directo-

ries of a data bank can be generically dynamically ascertained.

File systems and directories normally are not in a 1:1 relation to nodes and services. A

file system e.q. may be used by all nodes of a pool, directories on the other hand are only

available in dependency to specific services. The architecture of the FA file utilization

management offers therefore the possibility to specifically ascertain the data on frame,

node or service.

Introduction


The FA Agent takes over the data for every cycle from the detector and compresses the

results to a report cycle. The results are written in a collection file for every report cycle.

The collection files develop service-, node-, pool- and frame-specific.

Service-specific or node-specific information can only be generated by FA AppAgents.

During the collection of the data on the Control Node frame-specific storage data can be

collected.

For the administration of the storage consumption information, collections of variable

type are written:

Exampes:

Perf_FSUtil_Node

The collection develops for every node and contains a record with correspond-

ing timestamp of the report cycle to the report cycle for every file system to be

supervised.

Perf_DirUtil_Service

The collection develops for an SID service and contains a record with corre-

sponding timestamp of the report cycle to the report cycle for every directory or

file to be supervised.

Perf_FSUtil_Frame

The collection develops for every node and contains a record with correspond-

ing timestamp of the report cycle to the report cycle for every file system to be

supervised.

Introduction


3.9.1 File System Detector and Directory Detector

As of version 8.0 FA Agents have two additional detector components for the supervision

of file systems and directories.

1. File system detector

2. Directory detector

The file system detector can supervise file systems or mount points and supplies the

existing and employed capacity of a file system. A file system corresponds to a mount

point in a Linux system.

The directory detector enables the monitoring of individual files or directory. The directory

detector ascertains the absolute quantity.

Absolute storage space z

Total capacity of the file system

Usage of file system and directory

Usage %

The delta to the last cycle kb%delta and additionally as value per second

The resource consumption or the footprint of the directory detector is distinctly higher

then the file system detector consumption. Therefore the directory detector can only be

used for individual directories and should not be configurated for all directories.

Interesting is for instance the supervision of files and directories where significant

increase is possible for example logfiles or database directories.

Variable cycle times can be defined for collecting of the file system data and directory

data:

Perfdata_SpaceUtil_FSCycleTime

specifies how often (in seconds) myAMC.FA checks the file system utilization for

performance data.

Perfdata_SpaceUtil_DirCycleTime

specifies how often (in seconds) myAMC.FA checks the directory utilization for

performance data.

The FA Agent is able to calculate from the raw values the increase rates through consi-

deration of several cycles and to dispatch alerts as soon as these and/or the fill degree of

the file system exceed specific values.

Values for the generation of the file monitoring alerts receives the FA Agent from the configuration file myAMC.FA-Check.xml in which limiting values for every supervised file

system can be configurated.

This function enables for example the supervision of the availability and the capacity of

the voIFF filesystem from FlexFrame.

Introduction


3.10 Domain Manager for Performance and File Sys-tem Collections

3.10.1 Data collection

The domain manager is an independent component of the FA suite. The domain manag-

er cyclically processes the collection files for the performance, capacity, accounting and

file utilization management. The FA domain manager is used on the Control Node.

To that purpose the FA Agents generate so called file system file collections and accoun-

ting collections each in the data directory of a pool. The following types are distinguished:

Collections for every node with the name pattern Perf_Node~<node_name>.prf.<number>.col

Collections for every servicegroup with the name pattern Perf_Group~<node_name>.prf.<number>.col

Collections for every service with the name pattern Perf_Service~<SrvType>~<SID>~<ID>.prf.<number>.col

Collections for the file system supervision Perf_FsUtil_Frame~< frame_name>.prg.<number>.col

Perf_FsUtil_Pool~<pool_name>.prg.<number>.col

Perf_FsUtil_Group~<group_name>.prg.<number>.col

Perf_FsUtil_Node~<node_name>.prg.<number>.col

Perf_FsUtil_System~<SID>.prg.<number>.col

Perf_FsUtil_Service~<SrvType>~<SID>~<ID>.prg.<number>.col

Collections for the directory and file monitoring Perf_DirUtil_Frame~< frame_name>.prg.<number>.col

Perf_DirUtil_Pool~<pool_name>.prg.<number>.col

Perf_DirUtil_Group~<group_name>.prg.<number>.col

Perf_DirUtil_Node~<node_name>.prg.<number>.col

Perf_DirUtil_System~<SID>.prg.<number>.col

Perf_DirUtil_Service~<SrvType>~<SID>~<ID>.prg.<number>.col

Number and size of the collections generated by the FA Agents can be adjusted. The

standard setup enables the generating of 10 collections each per service or node. With

that a ringbuffer of data which can be automatically reorganized by the agents develops.

The for the sizing required storage size can be calculated from the number of the nodes

and the size of the report cycle.

The parameter of the performance manager and the backup routine have to be adjusted

in such a way that the raw data can be securely further processed bevor being super-

scribed again by the agents.

Introduction


3.10.2 Data retentiontime and compression

The domain manager has the task to process new data and write it to the database.

However, the domain manager also monitors the retention time of the individual data in

the database. After expiration of the retention time he automatically deletes older data

sets from the database.

From Flex Frame Flex Frame Orchestrator V5.3 or V1.0 Domain Manager can also com-

press data cyclically. From the minute values, e.g. be used for online monitoring, which

can lead to very large amounts of data very quickly, is cyclically generated a compression

stage (eg hourly values). The amount of data is reduced by a factor of 60. From the min-

imum, maximum and average values based on minutes, the corresponding values are

calculated on an hourly basis. For this compression not the collets, but the data from the

repository are required. The erase cycles and compression cycles can be configured

independently of each other. Prerequisite for a higher compression level is, of course,

that the deletion of the data needed for the compression will be done at a later time.


4 FlexFrame Autonomy

The operation of SAP systems is becoming increasingly complex, the number of compo-

nents required is constantly rising.

Installation, configuration and operation of a distributed SAP installation consequently

involve considerable administrative effort. The demands on the systems change rapidly,

and it must be possible to expand an existing configuration of replace failed components

both quickly and flexibly.

Through the use of Autonomy Agents, FlexFrame enables the number of operator inter-

ventions to be reduced and availability to be increased. This chapter describes the appli-

cation scenarios for the FlexFrame Autonomy functions.

Installation and startup of the agents is described in chapter 2.

To permit active operation of a FlexFrame Autonomy installation, the FlexFrame Autono-

my Agents must run on the Application Nodes and a Control Agent on the active Control

Node.

Use of the Messenger component is optional and is required only for displaying and for-

warding events of the FA Agents and for integration into Enterprise Event Management

Systems.

The FlexFrame Autonomy Application Agent is used to monitor SAP central instances,

SAP application instances and database instances. In the event of a problem, so-called

self-repair mechanisms are used for these services. Execution of these self-repair mech-

anisms can be triggered locally or centrally. For each service/node these mechanisms

can be divided into the following categories:

Monitoring of a service

Restarting of a service if it was down

Rebooting of a node if a service could not be started again after one or more re-

starts

Switchover (automatic change) to another node if the reboot could not be per-

formed or was not successful

Detecting of START, STOP and maintenance situations

Control functions for displaying activities and statuses, sending mails and SMSs,

configurable in conjunction with time, contact and problem situation

Performance, capacity and accounting management

File system monitoring

FlexFrame Autonomy


4.1 FlexFrame Autonomy Reactions

FlexFrame Autonomy detects problems and decides autonomously on the reactions to be

implemented after evaluating rules which can be controlled via parameters. FlexFrame

Autonomy knows the following basic reactions:

Restart

Reboot

SwitchOver (internal / external)

These basic reactions, combined with pool creation, grouping, the service classes, and

the service priorities result in a large number of reaction scenarios.

4.1.1 Restart

The FA AppAgent restarts a service if a required subservice is down or no longer avai-

lable. In this case it checks whether the service is available again after the restart on the

basis of a configurable time interval. The restart is not performed if any service which

runs on the node has already triggered a reboot. Furthermore, failure of multiple sub-

services of a service leads to a restart within the configured time interval only until service

availability has been restored.

The number of restart attempts for restoring service availability can be configured. If the

number of parameterized restart attempts is 0, failure of the service results directly in a

reboot attempt. If the number of reboots permitted for the nodes is 0, a switchover is

initiated.

The restart reactions are not affected by pool creation, grouping or service classification.

4.1.2 Reboot

A node is rebooted if an monitored service has failed and could not be made available

again after the configured number of restart attempts, or if no restarts are allowed.

The autonomous reaction also evaluates the service priority of the service which is dis-

turbed. If multiple services are running simultaneously on the nodes, it will be checked

whether services with higher priority are still running. If this is the case, the reboot is not

performed.

4.1.3 SwitchOver

A SwitchOver always leads to all the monitored services of a node being moved to an-

other server node.

The decision to move to another node can be taken locally by the FlexFrame Autonomy

Agent ( internal SwitchOver), or by the Control Agent on one of the Control Nodes (exter-

nal SwitchOver).

FlexFrame Autonomy


4.1.3.1 General Rules

Takeover in the event of a node failure is implemented using what is termed an applicant

rule. The applicant rule states that each spare node may apply to take over the services

of a failed node.

Pool creation, grouping and service classes permit new switchover scenarios which can

satisfy different availability requirements depending on the parameterization.

This results in the following scenarios:

Pool-dependent switchover

Group-dependent switchover

Service-prioritized switchover

The failure of a node is only reacted to within a virtual FlexFrame pool.

Groups can be defined within a virtual FlexFrame pool. The applicant rule states that a

node only issues an application when a node in its own group fails.

The granularity of the reaction to a system failure can be further refined by prioritizing

individual services.

The applicant rule states that in the event of simultaneous failure of multiple services, the

application is first issued for the switchover file (testament) of the service with the higher

priority. Only if all higher-priority services have been taken over by another node and free

spare nodes exist do these apply for the switchover files of lower-priority services which

still need to be taken over.

When services with priority 0 fail, no applications are made by spare nodes. This pre-

vents spare nodes being used up by the failure of unimportant test systems.

The parameter file also contains a “minimum priority” parameter. This parameter provides

a very simple way to define, for example, that spare nodes only apply to take over the

services of a node if none of the failed services has a lower priority than that entered

there.

In conjunction with the basic rule “by default all services have priority 1“, a lower priority

can be configured for individual services, thus providing a simple way to prevent valuable

spare nodes being used up by the failure of test systems.

4.1.3.2 Internal SwitchOver

In the case of an internal SwitchOver the Application Agent recognizes that a service is

down and cannot (or depending on the configuration may not) be restored using a restart

or reboot. The FlexFrame Autonomy Agent then initiates an internal SwitchOver.

The autonomous reaction also evaluates the service priority of the service which is dis-

turbed. If multiple services are running simultaneously on the nodes, it will be checked

whether services with higher priority are still running. If this is the case, the internal

SwitchOver is not performed.

FlexFrame Autonomy


The actual takeover by another node begins with the transfer message. Only spare nodes

can apply and take over these services. The node which takes over control starts the

required services. If, after the maximum switchover time, the FA AppAgent on the system

that is to take over control is not able to start the services, it reports this by means of an

SNMP trap. The switchover is aborted and must be processed further by the administra-

tor.

4.1.3.3 External SwitchOver

In contrast to the internal SwitchOver, the external SwitchOver is detected and initiated

by the Control Agent on a Control Node. This is required if the system is showing no sign

of life or can no longer be reached in the network. Reachability is tested using Ping or

SSH tests. The user decides whether to perform only Ping, only SSH or both kinds of

test. Additionally the Ping requests may be configured for client LAN, server LAN, and/or

storage LAN interfaces.

The takeover is performed in the same way as for the internal SwitchOver.

In order to enable user-specific actions before or after a node was powered down, the

CtrlAgent calls hook scripts, which may be customized by the user. The scripts are pro-

vided with the return code of the previously executed action.

Pre-PowerOff hook script: Called with return code 0 as argument, as there was no previ-

ously executed action.

Post-PowerOff hook: Called with the return code of the Pre-PowerOff hook script (if it

failed, i.e. the return code was != 0) or with the return code of the power off script.

If the configuration value IgnorePoffHookResult is set to true, the return codes of

the hook scripts are ignored. If set to false, they are used as hints on how to proceed in

case of errors: if the Pre-PowerOff hook script returns a value != 0, power off will not be

performed, if the Pre-PowerOff hook script returns a value != 0, the switchover will not

proceed. This enables the user to customize the external switch over and power off pro-

cesses based on additional information or rules or to perform additional actions, e.g.

mounting SAN devices.

FlexFrame Autonomy


4.1.4 Maintenance

The autonomous functions and reactions of the FA Agents can be disabled for individual

services by calling a maintenance script. This is always required when application in-

stances are to be started and stopped without autonomous reactions.

For FlexFrame Version >= V5.0A:

A service is set to nowatch using the following scripts on the relevant Application Node:

ff_service.sh -s <SID> -t <SERVICETYPY -a nowatch

ff_service.sh -s ml5 -t db -a nowatch

...

A service is reincluded in monitoring using the following scripts on the relevant Applica-

tion Node:

ff_service.sh -s <SID> -t <SERVICETYPY -a watch

ff_service.sh -s ml5 -t db -a watch

...

4.2 Self-Repair Strategies

In terms of the strategy for restoring a failed service, a distinction must be made between

the following failures:

Service failure

Node failure

A detailed description of the procedure for the subsequent autonomous reaction was

provided in the preceding chapter.

4.2.1 Self-Repair in the Event of a Service Failure

If a service failure occurs, this is detected by the myAMC.FA_AppAgent and an attempt

is made to make the service available again using the following autonomous reactions

and their escalations:

Restart of the service

Reboot of the node

SwitchOver (internal)

FlexFrame Autonomy


Implementation and the number of the above-mentioned autonomous reactions and the

escalations can be affected by the configuration.

4.2.2 Self-Repair in the Event of a Node Failure

If a node failure occurs, this is detected by the myAMC.FA_CtrlAgent and an attempt is

made to make the service available again using the following autonomous reactions:

SwitchOver (external)

4.2.3 Takeover by a Spare Node (TakeOver)

The standard rule in the FlexFrame concept for taking over the services of a failed node

is to have them taken over by a spare node.

Every Application Node in a standard FlexFrame installation on which an FA AppAgent is

running and none of the monitored services exists is automatically a spare node.

If a switchover is started as a result of a node failure or escalation of a service failure, all

spare nodes apply to take over the services. The quickest node in the application proce-

dure is chosen and takes over the tasks.

4.2.4 Multi Node Failure

4.2.4.1 Common

The simultaneous failure of multiple systems or nodes is called “Multi Node Failure“. This

indicates a different kind of failure than a single node or system failure, where the cause

is usually more complex. As of version V30A10, the FA Agents offer support for the au-

tomatic detection of Multi Node Failures with different reactions and additional alarms.

This allows the recognition of failure states, which require the attention and decision of an

administrator. Several new parameters allow the modification of the usual behaviour, like

delaying or skipping reactions. Additionally a set of new alarms triggered by user-

configurable indicators inform the administrator in case of of a multi node failure, so he

may take apprioriate actions. The configuration of these indicators can be performed per

pool.

There are two different Multi Node Failure scenarios:

1. Simultaneous failure of multiple nodes, systems or services, e.g. due to a power

outage in a blade cabinet, which shows an affect within a short period of time

(e.g. one minute)

2. Failure of several nodes, systems or services, within a specific timerange, which

is bigger than the one specified above (e.g. one hour)

These scenarios are called “ShortTime Failure” and “LongTime Failure”.

FlexFrame Autonomy


The CtrlAgent keeps a list of all failures, with each entry containing node name and

timestamp. If the number of entries within a scenario-specific timerange exceeds the limit,

the CtrlAgent assumes a Multi Node Failure.

4.2.4.2 Case 1: ShortTime Failure

Simultaneous failure of multiple nodes, systems or services, e.g. due to a power outage

in a blade cabinet, which shows an affect within a short period of time (e.g. one minute).

MultiNodeFailure_ShortTime_FailureCount

Specifies the number of failures within a certain time range, which leads to a Multi

Node Failure state.

MultiNodeFailure_ShortTime_FailureTime

Specifies the time range (in seconds) to be used for failure aggregation.

MultiNodeFailure_ShortTime_ReactionDelay

Specifies a delay time (in seconds) before the CtrlAgent reacts on failures.

MultiNodeFailure_ShortTime_ReactionAction (for future use)

Specifies a reaction different from the normal modus of operation.

In case of a Short Time Multi Node Failure the CtrlAgent sends an emergency alarm.

Additionally the usual autonomous reactions can be delayed or skipped (by setting

MultiNodeFailure_ShortTime_ReactionDelay to a very big value).

4.2.4.3 Case 2: LongTime Failure

Failure of several nodes, systems or services, within a specific timerange, which is bigger

than the one specified above (e.g. one hour).

MultiNodeFailure_LongTime_FailureCount

Specifies the number of failures within a certain time range, which leads to a Multi

Node Failure state.

MultiNodeFailure_LongTime_FailureTime

Specifies the time range (in seconds) to be used for failure aggregation.

MultiNodeFailure_LongTime_ReactionDelay

Specifies a delay time (in seconds) before the CtrlAgent reacts on failures.

MultiNodeFailure_LongTime_ReactionAction (for future use)

Specifies a reaction different from the normal modus of operation.

In case of a Long Time Multi Node Failure the CtrlAgent sends an emergency alarm.

Additionally the usual autonomous reactions can be delayed or skipped (by setting

MultiNodeFailure_LongTime_ReactionDelay to a very big value).

FlexFrame Autonomy


4.3 Escalation with partner service

There are special rules for some service types which have a partner-service.

There are differences if the partners run at the same or at another node as well as the

behavior at escalation and SwitchOver.

There are special rules for the following service-types:

• SCS / ASCS in conjunction with ERS

• Service with assigned SMD-service

4.3.1 SCS / ASCS in conjunction with ERS

For an SCS or ASCS service there is a replicated enqueue service (ERS) on which the

enqueue table is replicated.

The FA-Agent has to check is the replication was active.

Therefore the status of replication will be detected with the script (‚my-

AMC.DetTestEnsmon.sh‘).

Possible results:

# no replication service configured

ENSMON_RESULT="replication is disabled in server"

# replication service configured, but no replica running

#ENSMON_RESULT="replication is enabled in server"

# replication service configured, replica running

#ENSMON_RESULT="replication is enabled in server\nReplication is

active"

Only if the replication ist enabled and acrive, there is a special rule as VIP-service for

SCS/ASCS.

Up to now it was usual run the SCS / ASCS and the ERS at different nodes, so in cause

of failure there are special scenarios for TakeOver.

By now it is allowed to run SCS/ASCS and ERS at the same node.

These involved adapted rules for escalation.

4.3.1.1 SCS/ASCS with partner ERS at various nodes

Aim:

FlexFrame Autonomy


If the SCS/ASCS fails, this service must be restarted on that node where the associated

ERS is running.

The SCS/ASCS takes over the enqueue table in shared memory and stops the ERS.

Once the ERS is stopped, the testament is published and, with the autonomy scenarios

for internal SwitchOver, the ERS gets a new node and starts up. So if the SCS/ASCS

fails again, there is another ERS for a new takeover scenario.

For switchover of SCS/ASCS there was written a service based VIP-testament, which

contains only the SCS/ASCS and may only take over from the node running the associ-

ated ERS.

For switchover of ERS there was written a service based testament, which contains only

the ERS. For this TakeOver the normal takeover rules being valid.

Sequence of steps:

Node A Node B Node C

SCS/ASCS

running

ERS

running

SCS/ASCS

fail

SCS/ASCS

VIP-Testament

TakeOver

ERS

Switch

ERS-

Testament

TakeOver

These behavior with the VIP-testament will be done if configured both „Node_SwitchOverTyp=node“ and „Node_SwitchOverTyp=service“.

This scenario works with one or more replicated enqueue services in one system.

4.3.1.2 SCS/ASCS with partner ERS at same node

If SCS/ASCS and associated ERS are running at the same node, there is no shared

memory at another node which can be taken over.

Therefore the SCS/ASCS and ERS will be written in a common SwitchOver-file and will

be taken over to the same node.

These common switch of the partner services will be done if configured both

„Node_SwitchOverTyp=node“ and „Node_SwitchOverTyp=service“.

4.3.2 Service with assigned SMD-service

Service based SMD:

FlexFrame Autonomy


If a service has an assigned SMD running at the same node, in case of failure the ser-

vices should be switch to the same node.

Therefore the service and the assigned SMD will be written in a common SwitchOver-file

and will be taken over to the same node.

These common switch of the partner services will be done if configured both

„Node_SwitchOverTyp=node“ and „Node_SwitchOverTyp=service“.

Host based SMD:

If the SMD is host based there is no fix mapping to the service.

If a host based SMD is the only running service at a node, the myAMC.FA will be recog-

nized the node as spare node.

During the myAMC.FA reactions Restart and Reboot the SMD will be handled as every

other service.

FlexFrame Autonomy


4.4 Takeover Rules

4.4.1 Overview

Rule based high availability for nodes and services is performed by evaluating rule sets,

which control the take over of services from a failed node. They consist of qualification

rules, take over strategy and take over rules.

The qualification rules specifies, which nodes may apply for the services of a failed node.

The take over strategy defines the conflict resolution mode to be used, when more than

one node applies for a node testament.

The take over rules controls the actual take over, i.e. service start order and possibly

service displacement or replacement.

The parameters for this TakeOver rules always apply to the entire pool (Up Flexframe

V5.2).

From FlexFrame V5.3 or FlexFrame Orchestrator V1.0 these rules can be defined differ-

ently in each group of a pool.

In the default case, there is a set of rules that applies to the entire pool. In addition,

group-specific rule sets can be configured for one or more groups. For all groups for

which there is no separate set of rules, the default rule applies.

4.4.2 TakeOver Strategy

4.4.2.1 Overview

The qualification rules specifies, which nodes may apply for the services of a failed node.

When performing a switchover, all nodes may apply for take over of the failed node’s

services by taking part in an auction. As long as the auction lasts, all nodes, which

match the requirements as specified in the failed node’s testament may apply. When it is

finished, the take over strategy is used to decide which node won the auction.

4.4.2.2 FirstFit

FirstFit specifies that the first node, which applied for a testament, is the winner. This

is the default strategy.

4.4.2.3 LowPrioFit

As of version V30A10, the FA Agents provide a new strategy: LowPrioFit. The applica-

tion node containing the services with the lowest priority wins the auction. It therefore has

the best chance to replace or displace some running services in order to take over the

failed services.

FlexFrame Autonomy


By definition a spare node is considered to have the lowest priority, so it will win an auc-

tion over a node with running services. A node with only services of priority 0 will win over

a node with services of priority 1 and higher and so on.

This strategy can be used as an alternative to FirstFit. This changes only the behaviour of

the new take over rules: add rule, replace rule and substitution rule. If only the spare

node rule is used, the behaviour is the same as with the FirstFit strategy, because all

spare nodes have the same priority and the first one wins the auction.

4.4.3 TakeOver Rule

4.4.3.1 Overview

In version 3 and higher, the FA Agents offer the option of configuring various takeover

rules. It is now possible to replace or supplement the previously available spare node rule

with further alternatives.

Generally it is possible to differentiate between a static takeover rule and a dynamic

takeover rule.

These takeover rules can be applied not only for nodes but also for service-based testa-

ments. With node-based testaments it is evaluated in each case on the basis of the high-

est priority in the testament or the highest priority of a current service on an Application

Node.

4.4.3.2 Takeover Rules

The takeover rule is referred to as static if one of the possible takeover rules is param-

eterized. The spare node rule available until version 3 is a static takeover rule.

With version 3.0 of the agents the following static parameterized takeover rules will be

available:

Spare node rule

Add rule

Replace rule

Substitution rule

Dynamic

These rules allow very granular reaction to the breakdown of Application Nodes and

services. It is no longer necessary to always keep spare nodes on hand in case of a

breakdown of services for high availability.This function can now also be performed by

Application Nodes which already possess services.

4.4.3.2.1 Takeover through Spare Nodes (TakeOver)

The takeover by way of a spare node is the standard rule in the FlexFrame concept for

taking over the services of the defective node.

FlexFrame Autonomy


Every Application Node in a standard FlexFrame installation on which the FA AppAgent

runs and on which none of the controlled services exist, is automatically a spare node.

If, through the breakdown of a node or the escalation of a service disturbance, switchover

occurs, all spare nodes apply for the takeover of the services. The quickest node in the

application procedure receives the job and will take over those services. The application

takes place at group level.

In a default configuration, the spare node rule always applies.

4.4.3.2.2 Add Rule

With this rule all spare nodes of a group can apply but so too can all nodes which still

have sufficient workload reserves. The add rule only uses a node if the priority of the

services in the testament is equal to or higher than the priority already running on the

node. This prevents a high-priority node from taking over lower-priority services.

With add –rule only the nodes which possess services with lower or equal priority

apply (from pool, group).

If (OWN_prio_max >= SWO_prio_max ) apply

In addition to the already running services, the services that have been taken over are

started.

Thus the add rule is employed if running services are supposed to be stopped by the

takeover rule. This normally gives rise to a performance disadvantage after the taking

over of the services for the services taken over as well as the services already running on

the Application Node.

The add rule expects for the services a configured maximum SAPS value reserved for a

service. The node only applies if its SAPS workload suffices for the operation of the al-

ready running services and the services taken over.

4.4.3.2.3 Replace Rule

This rule enables all spare nodes of a group to apply, but also all other nodes on which

services with lower priority run. The replace rule only uses a node if the priority of the

services in the testament is higher than the priority of the services already running.

If ( OWN_prio_max > SWO_prio_max ) apply

Attention: Prio 1 is the highest priority: numbers > 1 decrease in priority.

Services which are already running are displaced. The adopted services are started.

A displaced service is transferred via the internal SwitchOver.

The other nodes of the group apply according to the application procedure and the ser-

vices can again be made available for other nodes, provided that lower-priority nodes still

exist within the group.

FlexFrame Autonomy


This rule defines which services may be stopped within the context of a takeover scenario

so that higher-priority services can be adopted and can possibly be made available on

another node.

4.4.3.2.4 Substitution Rule

This rule enables all spare nodes of a group to apply, but also all other nodes on which

lower-priority services run.

With the substitution rule, only nodes (from pool, group) possessing services with a lower

priority than the high priority of the switchover file apply.

If ( OWN_prio_max > SWO_prio_max ) apply

Attention: Prio 1 is the highest priority numbers > 1 decrease in priority

Services which are already running are stopped according to the rules of the stop hierar-

chy. The adopted services are started.

FlexFrame Autonomy


4.4.3.3 Dynamic TakeOver Rules

4.4.3.3.1 Overview

In the static mode only one of the possible takeover rules can count. With the parameter dynamic as static takeover rule the decision which rule ist used is made dynamicaliy,

depending on the priorities of the services on the Application Nodes and the highest prior-

ity of the defective services. For this purpose, the priorities table exists in the parameter

file, which is only used in the case of a static rule dynamically parameterized.

The dynamic takeover rule allows disjunct as well as overlapping priority domains be-

tween the spare node rule and one of the other static takeover rules. For the takeover rules add rule, replace rule and substitution rule, the priority domains must defi-

nitely belong to a rule. If a priority domain is defective then the first fitting is used.

The dynamic takeover rules are best understood from the view of a service on one of the

Application Nodes.

The add rule replace rule and the substitution rule can be seen from the view of a running

service on an Application Node as escalation stages which, within the context of a takeo-

ver scenario, increasingly handicap it unless it can avoid impairment through a higher

priority.

As soon as a testament is published for takeover, all application nodes of the group

check if they can improve their status (services with higher priority). An Application Node

applies if, on successful application, an improvement to its service priority status is possi-

ble.

The successful applicant determines the taking over dynamically according to the para-

meterized takeover rule. The rule results from the evaluation of the dynamic takeover

table and the highest priority of the running services.

From this view, naturally the spare node has a special role, as it has no services it could

handicap. It has the function of protecting its resources for special challenges. The pa-

rameters registered for the spare node result in principle in another evaluation. The pa-

rameters determine the priority domains in which the spare node may apply.The priority

domain for which a spare node is responsible can overlap with the priority domains of the

other rules. A spare node now no longer applies for defective services, as was the case

with simple spare node rules, but only for services in the priority domain. This prevents

spare nodes from beingwasted on operating defective lower priority services.

The rules are called dynamic takeover rules if, depending on the highest service priority

that an application server already has, it is dynamically decided according to which rule

an Application Node takes part in the application procedure on breakdown of other Appli-

cation Nodes or services .The decision as to which rule is applied depends on the ser-

vices running on an Application Node at that time.

The dynamic takeover rules are supposed to ensure that a node is always available for

the highest priority services. Lower priority services can be operated in parallel. On

FlexFrame Autonomy


breakdown of higher priority systems, those (the lower priority services) are displaced or

replaced.

The dynamic rules also have the function of reserving valuable spare nodes for high

priority services. This allows the use of different workloads and priorities in a FlexFrame

pool service. In case of hardware breakdowns very different disturbance scenarios, it is

now possible to use the remaining hardware optimally.

The application itself does not change. Of all the Application Nodes that applied, the one

which registered first wins.This is determined by the configuration parameter qualification

rule first fit.

The rules to be applied are parametered through a table with the following structure:

Spare node rule Prio >= 1 < 2

Add rule Prio > = 2 < 4

Replace rule Prio >= 4 < 6

Substitution rule Prio >= 6

The interpretation of this table depends on the dynamic evaluation implemented by an

Application Node for itself. An Application Node without services is a spare node. A spare

node applies only for node or service testaments with a service belonging to a priority

domain defined for spare nodes. This rule enables the reservation of spare nodes for high

priority services, without them being lost through lower priority services. This is shown by

the example with the following parametering for the spare node rule in the dynamic

takeover table.

Spare node rule Prio >= 1 < = 2

The spare nodes only apply for defective services which have a priority of 1 or 2. This

rule allows the restriction of the use of spare nodes on high-priority services.

Spare node rule Prio >=1 <= 2 exclusive

The additional attribute exclusive prevents Application Nodes which already possess

services from applying for services with the priority 1 or 2.

For Application Nodes which are not spare nodes, the rule for application is a bit more

complex. Here, two different dynamic influencing factors are at work: the highest priority

of a service in a testament for which it is possible to apply, and the highest priority of a

service already running on the Application Node.

For the application rule, the principle applies that an Application Node does not apply for

new services if this would cause already-running services with equal or higher priority to

be handicapped.

Example of a dynamic takeover rule:

FlexFrame Autonomy


Individual dynamic switchover rules per pool

1

1

2

2

Spare

3

4

Pool n

Group 1

Group 2

Sparenode >=1 < 2

Add rule >= 2 < 3

Replace rule >= 3 < 4

Substitute rule >= 4

Testament

Prio 2

5000 SAPs

2

2

3

4

Candidate with add rule

Candidate with repluce rule

Candidate with substitution rule

Group specific application process

FlexFrame Autonomy


4.4.3.3.2 Dynamic Example

In the Dynamic rule the highest priority of services in the switchover file and the highest

priority of the services at the own node will be used for further decision criteria.

For this, there is a further configuration:

Testament High Prio

TakeOver rule (min) (max)

SpareNode 1 4

Own High Prio

(min) (max)

Add 3 4

Replace 5 6

Substitute 8 20

Additionally there is the setting Dyn_Spare_exclusive, which means that the priority

range for SpareNode is exclusively reserved for the spare nodes.

The table above means the following:

High Prio in testament == 1 to 4:

Dyn_Spare_exclusive = true: Only spare nodes may apply.

Dyn_Spare_exclusive = false: All nodes may apply.

High-Prio in testament > 4:

Spare nodes may not apply.

All nodes with Own High Prio 3-4, 5-6, and 8-20 may apply.

All nodes with Own High Prio 1, 2, 7, and >20 may not apply.

Own High-Prio == 3 or 4: The TakeOver rule Add will be used.

Own High-Prio == 5 or 6: The TakeOver rule Replace will be used.

Own High-Prio == 7 to 20: The TakeOver rule Substitute will be used.

FlexFrame Autonomy


4.4.4 Spare Nodes from the Adminpool (pool-independent spare node)

As of version V8.0 the agents on spare nodes can be requested from another pool. The

CtrlAgent of a pool monitors the takeover of failed nodes and services. Are no applicants

for failed nodes available can the CtrlAgent request spare nodes from the admin pool if it

is configured

A CtrlAgent is able to recognize through evaluation of its own pool name and the config-

urated parameter, which ruleset is needed for its operation

The following configuration parameters are used:

CtrlAgentRole

Defines the role of the CtrlAgent

Normal Normal function of this CtrlAgent

(don't work with a caretaker of the spare pool). WorkWithSparePool CtrlAgent work with a caretaker of the spare pool

SparePoolName

Name of the spare pool

<name> Name of the spare pool

NONE No spare pool available

SpareGroup

Name of the spare node group

<name> Name of the spare group

NONE No spare group available

The following parameters for the pool-overlapping spare node are also required:

SpareRequestMoveTimeout

Timeout in seconds for move a node from spare pool to work pool.

SpareNodePowercycleTime

Time in minutes how often the spare nodes in caretaker will be powered ON and

OFF. Values <= 1 (60sec) are not allowed and switched of this functionality.

SpareRequestMoveNodeMaxRetry

Max retries to call move-node script in case of error (LDAP-locks).

SpareRequestMoveNodeRetryDelay

Delay-Time (in seconds) between retries the call of move-node sript.

4.4.4.1 Productive Pool

Productive pools are all those pools whose poolname does not correspond with the con-

figurated spare pool name.

FlexFrame Autonomy


4.4.4.2 SparePool (Adminpool)

4.4.4.2.1 General

The spare pool is the pool whose pool name corresponds to the configurated spare pool

name.

An adminpool is a special pool whose nodes are not used actively. A special group inside

an adminpool is the group SpareGroup. The spare group holds all nodes which are

available as spare nodes for other nodes.

4.4.4.2.2 Ascertainment of the Node Information

The spare nodes are normally turned off and are only cyclically turned on and off by the

CtrlAgents. The cycle time is defined through a parameter.

Initially the spare nodes have to be configurated manually and be available for at least

one detection cycle in the pool. With that the spare nodes with their capacity data are

automatically recorded and registered in a spare node list.

The cyclic turn on of the spare nodes actualizes the list every time. Nodes which can´t be

reached or started during that are then not in the list anymore.

4.4.4.3 Sparenode Selection Rule

To replaice a spare node of the productive pool with a spare node of the admin pool the

CtrlAgent selects a suitable node with a configurateable ruleset. The node which fits the

best with the failed node is selected.

The following attributes are considered for the selection of a failed node:

Attribute Meaning Examples

CPU-HT States

(if hyperthreading is active)

-1: not ascertainable

0: no hyperthreading

1: hyperthreading active

0

CPU-arch Processor architecture i686 sun4us

CPU-cnt-core Number of CPU cores 1

CPU-cnt-phys Number of CPU bases 1

CPU-cnt-siblings Number of CPU cores per

base

1

CPU-frequency(MHz) CPU tact frequncy 2993

FlexFrame Autonomy


Attribute Meaning Examples

OS-typ Operating system Linux SunOS

OS-version Version of the operating sys-

tem

SUSE LINUX Enter-

prise Server 9 (i586),

VERSION = 9,

PATCHLEVEL = 3

5.8

cache-size(kB) Size of the CPU caches (in

KB)

1024

hasHba States, if the node has a HBA

(host adapter for SAN) (true

or false)

false

hbaType Typ of the HBA (host adapter

for SAN)

none unknown

is10GbES States, if the node has a 10

GBit/s Ethernet interface (true

or false)

false

location Position of the node deposited

in LDAP

default

mem-total(MB) Main storagesize (in MB) 1001

saps Specified SAPS value of the

node

1042

node-product-name CPU term Intel(R) Pentium(R)

4 CPU 3.00GHz

FJSV,

GPUZC-M

node-vendor CPU producer GenuineIntel Sun_Micros

ystems

The concrete values of the attributes of the spare nodes are determined from the LDAP

and direct dynamically from each node by the FA Agents of the admin pool. Some

attributes (hasHba, hbaType, is10GbES and location) are determined from the

LDAP.

4.4.4.3.1 Configuration

The selection rules are defined in the config directory of the admin pool in the file my-

AMC_FA_SelectionRules.xml. They are organised in rule sets. Per inquiry only one

rule set is used: firstly the control agent searches a rule set with the name of the inquiring

pool. If such a ruleset does not exist, the default rule set is used.

FlexFrame Autonomy


A rule set contains a lot of selection rules which have to be checked from the CtrlAgent

for every possible substitute node. As a result every candidate has a weight function. The

candidate with the highest weight function is transfered as a substitute node to the inquir-

ing pool.

Example of the rule file:

<?xml version="1.0" encoding="iso-8859-1"?>

<rules xmlns="myAMC/SelectRules/1.0" defaultSet="default">

<ruleset schema="default">

<select attribute="OS-typ" match="equal" required="true">

<select attribute="CPU-arch" match="equal"/>

</select>

<select attribute="hasHba" match="equalOrMore"/>

<select attribute="hasHba" match="more"

required="false" weight="0">

<select attribute="location" match="equal"/>

<select attribute="hbaType" match="equal" weight="2"/>

</select>

<select attribute="is10GbES" match="equalOrMore"/>

<select attribute="saps" match="range" weight="1"

lowerBound="20" upperBound="50" required="false">




lowerBound="10" upperBound="20" required="false"/>





</select>

</ruleset>

<ruleset schema="Pool1">



</ruleset>

<ruleset schema="Pool2">



</ruleset>

</rules>

4.4.4.3.2 Rule evaluation

A single selection rule compares an attribute of the reference node (the failed node) with

that of a substitute candidate. The comparison can ensue through different operators.If a

rule applies the substitute candidate receives a certain amount of points. A score is de-

FlexFrame Autonomy


termined for the substitute candidate through evaluation of all rules of the ruleset. With

this score the most suitabel node is selected: the node with the highest score “wins”.

The following table describes the possible comparison operators:

equal Check of equality of an attribute. If both values are numbers a numeric

comparison is implemented; otherwise a textual comparison is imple-

mented.

more If the value of the candidate is more than the reverence value the rule

applies. If both values are numbers a numeric comparison is imple-

mented, otherwise a textual comparison is implemented. With boolean

status values (true, false) true in relation to false means “more”.

less If the value of the candidate is less than the reverence value the rule

applies. If both values are numbers a numeric comparison is imple-

mented, otherwise a textual comparison is implemented. With boolean

status values (true, false) false in relation to true means “less”.

equalOrMore If the value of the candidate is more or equal than or to the reverence

value the rule applies. If both values are numbers a numeric compari-

son is implemented, otherwise a textual comparison is implemented.

With boolean status values (true, false) true in relation to false means

“more”

equalOrLess If the value of the candidate is less or equal than or to the reverence

value the rule applies.I If both values are numbers a numeric compari-

son is implemented, otherwise a textual comparison is implemented.

With boolean status values (true, false) false in relation to true means

“less”.

range Checks if the value of the candidate in relation to the reverence value is

situated inside a certain area. The area limits aregiven as percentual

deviation with the help of the parameters “upperBound” and “lower-

Bound”. Differs the value of the candidate maximally over the given

limits from the reverence value the rule applies.

This comparison operator is only convenient with numeric attributes

e.g. the SAPS value (attribute “saps”)

The selection rules can be staggered hierachically. Subordinate rules are only evaluated

if the higher rule applies.

Example:

Rule 1

Rule 1.1

Rule 1.1

FlexFrame Autonomy


Rule 1.2.1

Rule 2

Rule 3

Rule 3.1

Every selection rule has its own weight function (parameter weight). If a rule applies its

weight function is added to the score of the selected candidate. Its possible to distribute

negative weight functions if an attribute rather serves to prevent a selection.

If the parameter weight is not given, the weight function is, depending on its hierachy

level, automatically put to the following value:

1.Level: weight=5

2.Level: weight=3

And all further Levels: weight=1

A rule can be marked as necessary (parameter required) i.e. a candidate is seen as

not fitting if a necessary rule is not fulfilled. If the parameter required is not given rules

on the first hierachy level are automatically considered necessary and all others as opio-

nal i.e. if they are not fulfilled the candidate can still be selectet if the score is sufficient.

4.4.4.3.3 Test of the Selection Rules

A simple test of the selection rules can be exeuted with fa_selectionRuleTest.sh

(in the directory /opt/myAMC/FA_AppAgent). The script evaluates matching candidates

to a reference node and issues the most fitting candidate.

fa_selectionRuleTest.sh

{-n|--source-node} <nodename>

{-p|--source-pool} <poolname>

{-P|--source-livelist} <file>

{-t|--target-pool} <poolname>

{-T|--target-livelist} <file>

[{-f|--rule-file} <file>] [{-r|--rule-set} <name>]

[{-B|--pools-basedir} <directory>]

[{-l|--log-level} <level number>] [{-o|--print}]

[{-a|--trace}] [{-c|--candidates}] [{-w|--weight}]

{-n|--source-node} <nodename>

Name of source node (mandatory)

{-p|--source-pool} <poolname>

Name of source pool (required if source-livelist is not set)

{-P|--source-livelist} <file>

Name of source input file (required if source-pool is not set)

FlexFrame Autonomy


{-t|--target-pool} <poolname>

Name of target pool (required if target-livelist is not set)

{-T|--target-livelist} <file>

Name of target input file (required if target-pool is not set)

{-f/--rule-file} <file>

Name of the rule file to use (optional)

Default: myAMC_FA_SelectionRules.xml in the target pool

{-r/--rule-set} <name>

Name of rule set to evaluate (optional)

Default: ruleset as specified in rule file

{-B/--pools-basedir} <directory>

Base directory for pools (optional)

Default: /opt/myAMC/vFF

{-l/--log-level <level number>}

Log level (optional)

Default: 5

-v|--verbose

Increase log level (optional)

-o|--print

Print node lists (optional)

-a|--trace

Trace selection rule evaluation (optional)

-c|--candidates

Print replacement candidates (optional)

-w|--weight

Print results including their weight (optional)

The tool determines the data of the reference node ( given with -n) from the accompany-

ing pool (-p)and searchs the fitting candidate in another pool (-t), the admin pool.

If additional options –a, -c and –w are given the results of all individual evaluations are

given. With this the correct parametricaton can be checked.

Example with several nodes:

cn1 # fa_selectionRuleTest.sh -a -w -c -n sno2bx1 -p pool2 -t adminpool

node "sno2bx3" does not match

sno2bx1 [36]

sno2bx2 [36]

Example with tracing:

FlexFrame Autonomy


cn1 # fa_selectionRuleTest.sh -a -w -c -n sno1apl5p1 -p pool1 -t adminpool

evaluating node sno1apl5p1

object : attribute OS-typ matched and added weight 5

matched 'SunOS' against 'SunOS' using operator 'equal'

object : attribute CPU-arch matched and added weight 3

matched 'sun4u' against 'sun4u' using operator 'equal'

object : attribute hasHba matched and added weight 5

matched 'false' against 'false' using operator 'equalOrMore'

object : optional attribute hasHba did not match

matched 'false' against 'false' using operator 'more'

object : attribute is10GbES matched and added weight 5

matched 'true' against 'true' using operator 'equalOrMore'

object : attribute node-powervalue matched and added weight 1

matched '7705' against '7705' using operator 'range'







node "sno1apl5p1" matches with weight 36

evaluating node sno1pw1

object : attribute OS-typ matched and added weight 5

matched 'SunOS' against 'SunOS' using operator 'equal'

object : optional attribute CPU-arch did not match

matched 'sun4u' against 'sun4us' using operator 'equal'

object : attribute hasHba matched and added weight 5

matched 'false' against 'false' using operator 'equalOrMore'

object : optional attribute hasHba did not match

matched 'false' against 'false' using operator 'more'

object : attribute is10GbES did not match

matched 'true' against 'false' using operator 'equalOrMore'

node "sno1pw1" does not match

sno1apl5p1 [36]

FlexFrame Autonomy


4.5 Management of Performance, File System and Accounting

FA AppAgents and FA FrameAgents have the possibility of collecting performance data

and file system data and generate cyclical defined value tuple with them. The result tuple

are frame-, pool-, group-, node-, system- and service-specific. The result of every cycle is

stored in a collection file. Collection files are temporary data containers for the storing of

the data. A collection file enables the storing of data from several report cycles.

The collection files themselves are organised in a ring buffer which enables the ascer-

tainment of the number and size of the collection files in a sizing process.

4.5.1 Performance and Accounting Management

The performance- and accounting management is already prepared in the standard con-

figuration. The performance and accounting functions are therefore in form of colletfiles

directly at disposal in supportcases as long as the necessary licence has been acquired.

The domain manager is necessary for the further processing and storing of the raw data

in the data bank.

4.5.2 File System Utilization

FA AppAgents and FA FrameAgents are able to determine the data for the file system

utilization. A parameter file defines, which file systems and directories are monitored.

The file system monitoring as well as the writing of the corresponding collections is direct-

ly activated by entry of file systems or directories which have to be monitored in the pool

specific configuration file myAMC_FA_Checks.xml.

4.5.3 Monitoring Cycles

For the collection of file system data and directory data variable cycle times can be de-

fined:

Perfdata_SpaceUtil_FSCycleTime

specifies how often (in seconds) myAMC.FA checks the file system utilization for per-

formance data.

Perfdata_SpaceUtil_DirCycleTime

specifies how often (in seconds) myAMC.FA checks the directory utilization for per-

formance data.

FlexFrame Autonomy


4.6 Operating Mode

FlexFrame Autonomy has various operating modes. The operating modes enable the FA

Agents to be used for simple monitoring tasks with automatic alarms, through to semi or

fully autonomous operation of a service.

Particularly in the startup and learning phases, this flexible configurability permits succes-

sive replacement of manual interventions by autonomous reactions.

The following operating modes are provided to implement the reactions:

Event mode

Local reaction mode

Central reaction mode

The operating modes can be defined at service level. This enables the degree of auton-

omous functions to be configured on an individual basis in accordance with the priority

and importance of a service. Parameterization of these modes takes place using the

parameters

Service_MaxRestartNumber

Node_MaxRebootNumber

Node_MaxSwitchOverNumber

4.6.1 Event Mode

No autonomous functions are performed in event mode. The events are just reported to

an event console in the form of messages. The administrator can then decide whether

the reactions proposed by the Autonomy Agents are appropriate and then execute these

manually. This mode is particularly recommended in the introductory phase. For this

purpose the parameter variables for the number of permissible restarts, reboots and

switchovers is set to 0.

4.6.2 Local Reaction Mode

Local reaction mode is the standard operating mode for a FlexFrame installation.

The reactions for restart, reboot and switchover are activated for all services in the my-

AMC.FA.xml parameter file, and at the same time the number of reaction attempts for a

reaction type is defined.

4.6.3 Central Reaction Mode

In central reaction mode the reactions are not initiated by the FA AppAgents. Here each

of the local parameter variables for restart, reboot and switchover is set to 0. The reac-

tions are initiated from a central position.

FlexFrame Autonomy


A option is that the reactions are created externally on the basis of the traps sent and

forwarded to the FA AppAgents via the BlackBoard.

FlexFrame Autonomy


4.7 Autonomous Operation of a FlexFrame Infrastructure

In every application environment the user needs an option for starting and stopping new

application instances. This will be done with the Fujitsu FlexFrame scripts.

4.7.1 FlexFrame Autonomy and Fujitsu FlexFrame Scripts

The reactions and user interactions take place by direct use of the Fujitsu FlexFrame

scripts.

The Fujitsu FlexFrame scripts are responsible for starting and stopping the SAP instanc-

es. They permit the SAP instances to be visualized and also supply the information re-

quired for the Autonomous Agents to detect the user interactions.

FlexFrame Autonomy


4.8 FlexFrame Autonomy and User Interactions

The autonomous reactions and the user interactions influence the status of FlexFrame-

Autonomy. Status changes and user interactions are logged in the log files and, if so

configured, also sent as traps.

Status creation begins with starting up the agents and the evaluation of the parameteriza-

tion information.

When the FA AppAgents are started up they read the parameterization file and subse-

quently know their job and operating mode. The FA Agents send a startup trap when they

start up.

Service instances that are already running are automatically recognized and managed.

The most important commands, reactions and their events in the various startup situa-

tions of the FA Agent are described in the following.

Restart, no services running:

The FA AppAgent sends only a node startup alert but no alerts for services

which do not exist.

Restart of the FA AppAgent when services are running:

The agent reports its own restart and availability and the various SAP services

which are running.

After reboot:

The agent reports its own startup and availability and the successful startup of

each individual service.

After switchover:

The agent that takes over control starts the services described in the testament

and send events corresponding to these.

FlexFrame Autonomy


4.8.1 myAMC.FA_Agents: Starting/Stopping/Status

4.8.1.1 Starting the myAMC.FA_Agents Manually

When the myAMC.FA_Agents are started a Agent start trap is sent.

An FA AppAgent can be started manually using the following command:


/opt/myAMC/FA_AppAgent/myAMC.FA_AppAgent start

An FA CtrlAgent can be started manually using the following command:

/etc/init.d/myAMC.FA_CtrlAgent start

/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent start

or

/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent start <pool_name>

/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent start Cust_1

An FA FrameAgent can be started manually using the following command:

/etc/init.d/myAMC.FA_FrameAgent start

/opt/local/myAMC/FA_FameAgent/myAMC.FA_FrameAgent start

4.8.1.2 Stopping the myAMC.FA Agents Manually

When the myAMC.FA_Agents are stoped a Agent stop trap is sent.

An FA AppAgent can be stopped manually using the following command:

/etc/init.d/myAMC.FA_AppAgent stop

/opt/myAMC/FA_AppAgent/myAMC.FA_AppAgent stop

An FA CtrlAgent can be stopped manually using the following command:

/etc/init.d/myAMC.FA_CtrlAgent stop

/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent stop

or

/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent stop <pool_name>

/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent stop Cust_1

An FA FrameAgent can be stopped manually using the following command:

/etc/init.d/myAMC.FA_FrameAgent stop

/opt/local/myAMC/FA_FrameAgent/myAMC.FA_FrameAgent stop

FlexFrame Autonomy


4.8.1.3 Status of the myAMC.FA_Agents

The status of the FA AppAgents can be inquired using the following command:

/etc/init.d/myAMC.FA_AppAgent status

/opt/myAMC/FA_AppAgent/myAMC.FA_AppAgent status

The status of the FA CrtlAgents can be inquired using the following command:

/etc/init.d/myAMC.FA_CtrlAgent status

/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent status

or

/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent status <pool_name>

/opt/myAMC/FA_CtrlAgent/myAMC.FA_CtrlAgent status Cust_1

The status of the FA FrameAgents can be inquired using the following command:

/etc/init.d/myAMC.FA_FrameAgent status

/opt/local/myAMC/FA_FrameAgent/myAMC.FA_FrameAgent status

FlexFrame Autonomy


4.8.2 Starting/Stopping an SAP Instance

4.8.2.1 Starting an SAP Instance

A further SAP instance of the type DB, CI, APP, J, JC, SCS, ASCS, LC, ERS, … can be

started at any time. Use of the FlexFrame start scripts in a version which has been re-

leased for the FA Agents is mandatory.

An instance which is to be monitored via the Autonomy Agents is started using the follow-

ing script calls on the Application Node:


ff_service.sh -s <SID> -t <SERVICETYPY -a start

ff_service.sh -s ml5 -t db -a start

...

The startup of an instance is documented by the agents using the following traps:

ServiceIsStarting trap,

ServiceIsStarted trap,

or

ErrorStartingService trap

FlexFrame Autonomy


4.8.2.2 Stopping an SAP Instance

An active SAP instance of the type DB, CI, APP, J, JC, SCS, ASCS, LC, ERS, … can be

stopped at any time. Use of the FlexFrame stop scripts in a version which has been re-

leased for the myAMC.FA Agents is mandatory.

Service instances which are stopped while the myAMC.FA_AppAgent is running are

detected by the agent and acknowledged with corresponding SNMP traps. In this case

the FA Agent can use mechanisms integrated into the FlexFrame solution to distinguish

between an instance being intentionally stopped and a service or instance crashing.


ff_service.sh -s <SID> -t <SERVICETYP> -a stop

ff_service.sh -s ml5 -t db -a stop

...

The following traps are sent:

ServiceIsStopping trap,

ServiceHasStopped trap,

or

ErrorStoppingService trap

FlexFrame Autonomy


4.8.3 NoWatch/Watch an SAP Instance

The "nowatch" of a service is now persistent. Therefore, the "nowatch" is no more a ser-

vice-state, but an attribute of a servive.

The persistent "NoWatch" is stored in the following file:

/opt/myAMC/vFF/vFF_<POOLNAME>/data/FA/reboot/Persist_FA_Info.txt

If this file will be deleted, the persistent information will be lost and the services which

was before in file will be monitored.

That the "NoWatch" is stored as persistent information, a AppAgent must be running at

AN.

4.8.3.1 NoWatch an SAP Instance

An instance which is to be monitored via the Autonomy Agents is “NoWatch” using the

following script calls on the Application Node:


ff_service.sh -s <SID> -t <SERVICETYPY -a nowatch

ff_service.sh -s ml5 -t db -a nowatch

...

4.8.3.2 Watch an SAP Instance

An instance which is to be monitored via the Autonomy Agents is “Watch” using the fol-

lowing script calls on the Application Node:


ff_service.sh -s <SID> -t <SERVICETYP> -a watch

ff_service.sh -s ml5 -t db -a watch

...

FlexFrame Autonomy


4.9 Use cases

4.9.1 General

The FlexFrame Autonomy solution offers various configuration and application options.

Application and Control Agents share the tasks for implementing autonomy in a Flex-

Frame solution.

The tasks of the Control and Application Nodes vary according to their configuration.

Some typical autonomy scenarios for the most important applications are presented in

the following:

Passive monitoring of your instances (notification mode); no reactions take place

(event mode).

Control of the instances’ availability using active FlexFrame Autonomy Applica-

tion Agents (local reaction mode); the reactions are triggered by the FA Ap-

pAgent.

Control of the instances’ availability using passive FlexFrame Autonomy Applica-

tion Agents and active High-Autonomy Control Agents (central reaction mode);

the reactions are triggered by the FA CtrlAgent.

The settings for notifications and reactions can be configured independently.

FlexFrame Autonomy


4.9.2 Semi-autonomous operation

In certain situations it may make sense only to use notification functions of myAMC.FA

and initially to dispense with active intervention in the system. This scenario is practical,

for example, for using a central position to monitor various systems and analyze failure

frequencies.

4.9.2.1 Monitoring of Application Instances

In order to monitor application instances, an FA AppAgent must run on every agent which

is equipped with detectors for the application instance to be monitored and has been

configured for this. The monitoring and reaction can be configured individually for each

application instance.

The service-specific parameters are set in the configuration section of the services. A

configuration for the event mode can achieved by the following settings in the Configsection Services Default section:

Service_SendTraps : true

Service_EnableMonitoring : true

Node_MaxRebootNumber : 0

Node_MaxSwitchOverNumber : 0

Service_MaxRestartNumber : 0

Servcice_TrapSendDelayTime : 0 or greater

With this configuration no reactions take place for any service type. However, event mes-

sages are sent if services are not available or have failed.

The parameters can, for example, be set on a service-specific basis for the semi-

autonomous operation of services.

In the Configsection Services APP section:






Service_MaxRestartTime : 120

With these setttings for the APP service, up to three restart attempts are made to render

an application service available again after it has failed. No reactions are implemented for

the other service types CI and DB. However, event messages are sent if services are not

available or have failed.

FlexFrame Autonomy


4.9.3 Autonomy for Application Instances

To permit autonomous operation of applications, FlexFrame Autonomy provides the op-

tion of monitoring instances and reacting actively to the failure of a service. The type of

reaction depends on the configuration set.

The following parameters can be used for this purpose in the parameter file:






Service_TrapSendDelayTime : 0 or greater

Service_ReactionDelayTime : 0 or greater

MaxRebootTime : 120

4.9.3.1 Restart

Restarting a service is the first option Autonomy Agents have of reacting to a service

failure.

For this purpose a Service_MaxRestartNumber greater than 0 is specified and also

the Service_MaxRestartTime. The Service_MaxRestartTime is evaluated by the

FA AppAgent.


Service_MaxRest artTime : 240

The time by which the service must be available again is the Service_MaxRestartTime of this service.

A maximum of Service_MaxRestartNumber attempts are made to restore availability

through this reaction; escalation to the next escalation level then follows.

FlexFrame Autonomy


4.9.3.2 Reboot

Rebooting a node is a further option for an Autonomy Agent to react and the next escala-

tion level after a restart.

For this purpose a Node_MaxRebootNumber greater than 0 is specified and also the

Node_MaxRebootTime. The Node_MaxRebootTime here is evaluated by both the FA

AppAgent and the FA CtrlAgent. After the Node_MaxRebootTime has elapsed the Con-

trol Agent uses the Node_CheckAvailabilty command to check the availability of the

system. If the RebootNumber is set to two or three, this value is increased by the corre-

sponding factor.


MaxRebootTime : 120

The time by which a server and all its services must be available again is calculated by adding the Node_MaxRebootTime and the greatest Node_MaxRestartTime of all

services to be started on this server.

A maximum of Service_MaxRestartNumber attempts are made to restore availability

through this reaction; escalation to the next escalation level then follows.

4.9.3.3 Switchover

A distinction is made between the internal and external SwitchOvers.

4.9.3.3.1 Switchover intern

The internal SwitchOver of all a node’s services is a further reaction option for the Auto-

nomy Agent and the next escalation level after a reboot. An internal SwitchOver is initiat-

ed by a failed node which actively wishes to transfer its services. This is therefore a fur-

ther reaction option for the Autonomy Agent and the next escalation level after a reboot.

For this purpose a Node_MaxSwitchOverNumber greater than 0 is specified.


The time by which a server and all its services must be available again is calculated by

adding the Service_MaxRestartTime of all services to be started on this server.

4.9.3.3.2 Switchover extern

An external SwitchOver is initiated by a FA CtrlAgent which determines that an Applica-

tion Node is no longer working.

FlexFrame Autonomy


4.9.4 Management of Performance, Capacity and Account-ing

The mangement of performance, capacity and accounting of the FA Autonomy is an

optional function. The use of this component requires a standard installation for the FA

Agents and additionally the installation of the FA Domainmanager. The function has to be

activated in the configfile of the FA Agents.

Depending on the demands for mangement of performance, capacity and accounting the

report cycle and the size of the ringbuffer for the each performance collection type has to

be defined. Additionally the FA Domainmanager has to be installed and started on the

active Control Nnode. The FA Agents supply the performance data nodes and capacity

data nodes service and SAP system specific as is standard without having to use individ-

ual configurations.

The following example describes the default configuration. The default configuration sup-

plies the performance data nodes and capacity data nodes system specific and service

specific. Also the following groups are already predefined.

The groups supply a node specific view of the portions on the whole output consumption

of a node.

SAP

NONSAP

FA

Idle

4.9.5 File System Utilization

The management for the supervision of the fill state and the increase of file systems can

be activated through simple activation in the FA config file. The data for the file system

expenditure can be supplied by the FA AppAgents or the FA FrameAgents.

The activation of the filesystem expenditure activates the default configuration. The de-

fault configuration supplies throught the FA FrameAgent the supervision of the voIFF and

dispatches an alert if this is filled more than 90% or the increase rate inside a report cyle

is more than 5%.

The ascertainment of the fill state of a file system and its increase rate can be ascer-

tained with relativly minor time expense and footprint.

FlexFrame Autonomy


4.9.6 High Availability and Autonomy Rules

The myAMC.FA suite can be configured through a set of parameters for different high

availability and autonomy situations. In the following examples the interaction and the

different reaction scenarios of some parameters is shown.

The examples also show how the interplay of the parameters in special high availablity

situations an emergency event is triggered and how the service priority and the service

capacity requirements influence the availability in a failure scenario.

Possible application scenario:

In a data center, there are one or more spare nodes within a pool. In case of failure of an

application server spare nodes take over the services that are running on the failed appli-

cation server.

Simultaneous failure multiple servers, only the number of application nodes can be made

available again on a Sparenode as there are free spare nodes within the pool, where are

the failures. If more than two nodes have failed within 5 minutes, then is this to be report-

ed over an emergency event. If more than 4 nodes have failed within 2 days, then this

should also be reported over an emergency event to a central console.

In general, the rules for each pool can be parameterized by the other pools within a Flex

Frame installation independently.

The following parameters will be used:

Node_SwitchOverTyp: service or node (default = node):

Depending on the setting of this Paramter node or service based SwitchOver-Files

will be written.

EscalationType: Service or Node (default = Node):

Depending on the setting of this Paramter the escalation will be done for all services

on a node (restart reboot switchover), or the escalation will be done for a single

service (restart swtchover (single service)).

TakeOverStrategy: FirstFit or LowPrioFit (default = FirstFit):

Depending on the setting of this Paramters wins in a failure scenario, the server has

become the first candidate for the switchover file of the failed node or the server that

had the lowest priority of its services.

Pool independet spare (default = no pool independent spare):

If no spare nodes are available, in failure scenarios in a pool, so there is the possibil-

ity to define a Spare Pool. Then, when applying no Sparenode in the pool for a

switchover file, then tries to get a Sparenode from the spare pool.

FlexFrame Autonomy


Parameter must be set in Pool and Sparepool: CtrlAgentRole: WorkWithSparePool (default = Normal)

SparePoolName: <Name of SparePool> (default = NONE)

SpareGroupName: <Name of SpareGroup> (default = NONE)

TakeOverRule (default = SpareNode):

This parameter determines the type of the adoption of a switchover file.

possible settings: SpareNode: Only a spare node will Start the services from taken SwitchOver-File.

add (Add, addition): Additional start the services from taken SwitchOver-File.

replace (move): Running services will be stopped and a SwitchOver-File will be

written therefore, Start the services from taken SwitchOver-File. substitute (displacement): Running services will be stopped, start the services

from taken SwitchOver-File.

dynamic: Cause of prios there will be done one of the TakeOver-rules SpareNode,

add, replace, substitute, or none.

For dynamic rule the following parameters are important: Dyn_Spare_min, Dyn_Spare_max; Prio-range

Dyn_Add_min, Dyn_Add_max; Prio- range

Dyn_Replace_min, Dyn_Replace_max; Prio- range

Dyn_Substitute_min, Dyn_Substitute_max; Prio-Bereich

Dyn_Spare_exclusive: Switch for „exclusive for spare“

Vip Services: VIP Services are services for which there is a special takeover scenario

outside of the standard. That is, for example, the ERS (enque Replica service).

The High Availability Rules can be divided into static and dynamic rules.

FlexFrame Autonomy


4.9.6.1 High Availability: Static Rules

In the Static Rules that apply generally to any failure in the affected pool. The basic rules

Sparenode-rule, Replace rule, add-rule-rule and substitution of the static rules.

Example: On an application server to run multiple Appliaktionsservices.

Sparenode-rule:

P01 with ci (priority 2) and the associated database db P01 (priority 1) fails. In the pool

there is no spare nodes. Thus, there is no high availability for the canceled services be-

cause there is no free node for the acquisition of the failed service.

Replace-rule:

P01 with ci (priority 2) and the associated database db P01 (priority 1) fails. Each applica-

tion node in the same pool with lower service priorities applying for the acquisition of the

switchover file. Own services with worse priority (ie 3 and greater) are stopped and then

published as the switchover file.

4.9.6.2 High Availability: Dynamic Rules

Upon failure of Services, Dynamic Rule allows a differentiated use of available spare

nodes and free capacity on application nodes which are not yet fully utilized, or run on

those with lower priority services.

For this use the parameterization of service priority and a service associated capacity

(power value) is important. When not only the spare node rule is used, but one of the

other rules, individual prioritization of services is necessary.

FlexFrame Autonomy


4.9.6.3 High Availability: Examples

In the following examples we have a production system with P01 3 instances and a quali-

ty assurance system with 3 instances and a development system. The production system

has the highest priority, then comes the quality assurance system and the development

system with the lowest priority. The services are operated in a flexible frame with a pool,

which consists of 12 application nodes.

The application node Node_10, Node_11 and Node_12 are spare nodes, since no ser-

vices running on that node.

4.9.6.3.1 Example 1 (SpareNode)

System ID

Servicetyp Prio Power value

Applikation node

Pool Group Scenario Node- capacity

P01 DB 1 6000 Node_01 Pool_1 Default t 10000

P01 CI 1 3000 Node_02 Pool_1 Default t 10000

P01 App 3 2000 Node_03 Pool_1 Default t 10000

Q01 DB 10 3000 Node_04 Pool_1 Default 10000

Q01 CI 10 1000 Node_05 Pool_1 Default 10000

Q01 App 30 1000 Node_06 Pool_1 Default 10000

E01 DB 20 500 Node_07 Pool_1 Default 10000

E01 CI 20 500 Node_08 Pool_1 Default 10000

E01 App 40 500 Node_09 Pool_1 Default 10000

Node_10 Pool_1 Default 10000


Node_12 Pool_1 Default

Default parameter settings:

Parameter Value Value Options

Node_TakeOverRule SpareNode SpareNode,Addrule,Displacement, Supplementation, dynamic

Noed_SwitchOverTyp Node Node, Service

Take_OverStrategy FirstFit FirstFit, LowPrioFit

The failure of the 3 node with the productive instances of the SAP system P01 is as-

sumed in this scenario of the 3 spare nodes.

FlexFrame Autonomy


FlexFrame Autonomy


4.9.6.3.2 Example 2 (SpareNode)

System ID

Service- typ

Prio Power value

Applikation node


P01 DB 1 6000 Node_01 Pool_1 Default t+60 10000

P01 CI 1 3000 Node_02 Pool_1 Default 10000

P01 App 3 2000 Node_03 Pool_1 Default 10000




E01 DB 10 500 Node_07 Pool_1 Default t 10000

E01 CI 10 500 Node_08 Pool_1 Default t 10000

E01 App 10 500 Node_09 Pool_1 Default t 10000






Node_TakeOverRule SpareNode SpareNode,Addrule,Displacement, Supplementation, dynamic



In this scenario at time t the 3 nodes from the development system fails. The at time t

existing 3 spare nodes take over these services and they are available again. Cause of

the consumption of the existing spare capacity, there is at time t+x no high availability for

the failure of an productive services of the P01 system.

FlexFrame Autonomy


4.9.6.3.3 Example 3 (dynamic)

System ID

Service- typ

Prio Power value

Applikation node


P01 DB 1 6000 Node_01 Pool_1 Default 10000

P01 CI 1 3000 Node_02 Pool_1 Default 10000













Node_TakeOverRule dynamic SpareNode,Addrule,Displacement, Supplementation, dynamic



Additional parameters that are needed along with the dynamic Node_TakeOverRule:

Paramater Value

Dyn_Spare_min 1

Dyn_Spare_max 2

Dyn_Add_min 3

Dyn_Add_max 30

Dyn_Replace_min 100

Dyn_Replace_max 100

Dyn_Substitute_min 200

FlexFrame Autonomy


Dyn_Substitute_max 200

Dyn_Spare_exclusive true

By using the Node_TakeOverRule in dynamic mode, the spare nodes for the productive

system is protected because the spare nodes are used with these parameters only for

services with priority <= 2.

Upon failure of the Q or E systems, these are added by other systems. With add–rule

only the nodes which possess services with lower or equal priority apply (from pool,

group).

In our example the node with P instances would not accept any services from Q and E

systems and therefore do not affect their performance.

The example also shows that the failure of the node node_3, with the productive applica-

tion instance of the P01 system, is not the high availability of this service through the use

of an idea node, but by the add rule. I.e. this app instance would run after a failure sce-

nario on one node with other services.

Note:

If new SAP services added to a pool, these have default values first.

The priority assigned to a service and the power value can be clearly visualized with the

myAMC.FA ConfigGui, and be changed as needed directly in this view.

FlexFrame Autonomy


4.9.6.3.4 Example 4 (sercive specific reaction)

Failure scenarios ERS service

The enqueue replication service has a special role in a high-availability scenarios.

The ERS belongs to an SCS or ASCS service only and assumes the failure of the asso-

ciated SCS or ASCS its task and thus becomes the new SCS / ASCS. Thus, in a second

failure of the SCS / ASCS another ERS instance is available it must be started on another

server. Therefore, this ERS is made available again by switchover to another node.

In the following example the ERS will be available at the spare node Node_11, but in

dynamic mode it my also be available will add- or other rule.

System ID

Service- typ

Prio Power value

Applikation node


P01 DB 1 6000 Node_01 Pool_1 Default 10000

P01 SCS 1 3000 Node_02 Pool_1 Default t 10000


P01 ERS 3 2000 Node_03 Pool_1 Default t+1 10000










FlexFrame Autonomy


4.9.6.3.5 Example 5 (group based rules)

The rules for the autonomy and high availability (Sparenode, Add, Replace and Dynamic

Rule) usually always apply to the entire pool.

If this high availability rules are defined at the group level, the response for the same

service type can be very different depending on which group the service is running on.

Defining rules on group level which together with the dynamic rule a plurality of different

reaction scenarios for failure.

FlexFrame Autonomy


4.10 FA Work and Log Files

The functions of the FA Agents are documented in various files. These files may not be

changed manually as this can impair error free operation of the FA Agents or result in

errored reactions.

These files are created dynamically during ongoing operation. Deleting these files leads

to a status in which the Autonomous Agents reorganize themselves, and from this point

they reevaluate the situation from the current viewpoint without any previous knowledge.

Base directory: /opt/myAMC/

Base directory: /opt/local/myAMC/

A detailed description can be found in chapter 3.4 resp. 10.3.

FlexFrame Autonomy


4.11 Migration of FA Agent Versions on Pool Level

The FlexFrame Autonomy Agents offer a whole raft of strategies and functionalities for

installing and activating patches and new release versions for a wide range of security,

test and release scenarios.

The administrator can use the update and activation functionality provided by the agents

in line with his/her requirements. The following basic functions are available:

A Reading and observing update, patch and release notes

B Installation of a new FA Agent version parallel to an operating FA Agent version. All

data and configuration information for the operating FA Agent version are retained.

C Taking over of the configuration data for the new FA Agent version using the FA

migration tool

D Pool-by-pool configuration/parameterization and activation of the new FA Agent

version

E Testing of a new FA Agent version, e.g. in a separate test pool, if required by deac-

tivating the autonomous reactions for test operation

The following activities are required to install or update a FlexFrame Autonomy Agent

patch or a newer release version.

1. Reading of the update, patch and release notes and observation of any required

modifications and special features, in particular in the event of simultaneous updating

of FlexFrame and operating system versions and patches.

2. Installation of a patch or a new release version (FA CtrlAgent and FA AppAgent).

FlexFrame Autonomy


3. Parameterization/configuration, possibly using the FA migration tool.

3.1. Copying the parameters from the active agent version to the migration configu-

ration directory using the FA migration tool. Normally the following call is suffi-

cient to do this:

/opt/myAMC/FA_CtrlAgent/MGRTool.sh --target-release=<release>

--migrate-pool=<pool> --backup

This following call migrates the configuration of the pool1 pool to the version

V20K23 for the associated FA Agents.

/opt/myAMC/FA_CtrlAgent/MGRTool.sh --target-release=V20K23

--migrate-pool=pool1 --backup

Starting pool migration

pool: pool1

source release: V20K22

target release: V20K23

migration dir: /opt/myAMC//vFF/vFF_pool1/Migration.V20K23.2005-05-10

migration succeeded

see file /opt/myAMC/vFF//vFF_pool1/Migration.V20K23.2005-05-10/

MIGRATION-INSTRUCTIONS.txt

for details and installation instructions.

The modified files are written into a subdirectory and no current files are modi-

fied. In addition, the migration directory contains a backup of all current files. All

new/modified parameters are listed in the MIGRATION-INSTRUCTIONS.txt

file.

3.2. Testing, parameterizing and configuring the parameters taken over in the TrapTargets.xml, myAMC_FA_Groups.xml, myAMC_FA_Rules.xml,

myAMC_FA.xml, myAMC_FA_GUI.xml and myAMC_FA_SD_Sec.xml files.

3.3. Testing any new parameters and, if necessary, modifying the default values en-

tered.

3.4. Parameterizing and configuring of FlexFrame/operating system version de-

pendencies if the FlexFrame basis is updated at the same time.

3.5. Check the modifications made by the migration tool, according to the file

MIGRATION-INSTRUCTIONS.txt in result directory of migration.

FlexFrame Autonomy


4. Pool-specific deactivation of the active FA Agent..

4.1. Stopping the FA CtrlAgent for the pool whose agents are to be updated.

/etc/init.d/myAMC.FA_CtrlAgent stop <pool>

4.2. Stopping the FA AppAgents on all nodes of the pool whose agents are to be

updated.

/etc/init.d/myAMC.FA_AppAgent stop

5. Pool-specific activation of the new FA Agents.

5.1. Modifying the active agent version in the .info file in the associated pool di-

rectory. To do this the pool.release.current entry must be adapted ac-

cordingly. The syntax is VxxKyy. This syntax is mandatory.

control1:/ # cat /opt/myAMC/vFF/vFF_pool1/.info

# Version V20K17

pool.release.base=V20K17

pool.release.current=V20K23

This pool will use the FA Agents of the versions V20K23.

Alternately this file can be transferred from the migrated configuration to the

configuration directory:

cd /opt/myAMC/vFF/vFF_pool1

cp ./Migration.<version>.<date>/.info .

5.2. Transferring the migrated configuration to the configuration directory.

cd /opt/myAMC/vFF/vFF_pool1

cp ./Migration.<version>.<date>/config/* ./config

5.3. Starting the FA AppAgents on all Application Nodes of the updated pool.


5.4. Starting the FA CtrlAgent.

/etc/init.d/myAMC.FA_CtrlAgent start pool1

FlexFrame Autonomy


6. Checking the new active FA Agent version.

6.1. Checking the agent processes.

Output for the example of the pool pool1:

control1:/ # /etc/init.d/myAMC.FA_CtrlAgent status pool1

Status of myAMC.FA_CtrlAgent ( myAMC_FA_CtrlAgent ) in vFF='pool1' at

host 'control1' ...

root 17608 1 0 13:41 pts/5 00:00:00 ./myAMC_FA_CtrlAgent

vFF=pool1 -lf=/opt/myAMC/vFF/vFF_pool1/log/CtrlAgt/









control1:/ #control1:/ # echo $?

0

6.2. Checking agent messages at startup.

control1:/ # /etc/init.d/myAMC.FA_CtrlAgent start pool1

Found vFF='pool1'.

Checking the files for vFF='pool1'...

Starting myAMC.FA_CtrlAgent ( myAMC_FA_CtrlAgent ) Version ( V20K23 ) in

vFF='pool1' at host 'control1' ...

6.3. Diagnosis and checking that the shown data is correct

6.4. Performing FlexFrame Autonomy tests (restart, reboot, etc.).

Steps 1, 2 and 3 can take place while FlexFrame Autonomy is active. The FA Autonomy

functions are not available only for a brief period between deactivation of the active agent

version and activation of the new agent version.

Note that only version-compatible FA CtrlAgents, FA AppAgents and FlexFrame versions

can be used. Compatibility of the agent versions with various FlexFrame versions results

in dependencies which must be taken into account.

FlexFrame Autonomy


4.12 The FA Migration Tool

The FA migration tool is used to migrate configurations of a selected pool to and from a

particular FA Agent version. The FA migration tool also enables you to merge configura-

tion files.

4.12.1 Pool Mode

Pool mode generates a migrated configuration in the subdirectory Migration.<version>_<timestamp>, including the backup of the current files. To

enable the migrated configuration to be used it must be copied into the relevant configu-

ration directory of the pool concerned.

Required / useful parameters:

-p/--migrate-pool=<pool>

-r/--target-release=<release>

-b/--backup

[-V/]--verbose]

[-d/--pools-basedir=<dir>] (optional)

[-c/--clean] optional

[-s/--source-release=<release>] (optional)

Example:

MGRTool.sh --migrate-pool=<pool> --target-release=<release> --backup

4.12.2 File Mode

File mode merges two files which are in myAMC config format. The two files are defined with the parameters merge-file and template.

File mode can only be used on files which are in myAMC config format (e.g. my-

AMC_FA.xml). The myAMC_Pools.xml and myAMC_FA_Groups.xml files are not in

this format. These files can therefore not be migrated using file mode, but only in pool

mode.

Required / useful parameters:

-m/--merge-file=<file>

-t/--template=<template>

-o/--out-file=<file>

[-V/--verbose]

[-c/--clean] (optional)

FlexFrame Autonomy


Example:

MGRTool.sh –-merge-file=File1.xml --template=File1-default.xml

--out-file=File-out.xml

4.12.3 Usage

Usage:

/opt/myAMC/FA_CtrlAgent/MGRTool.sh --help

myAMC_FA_MGR_Tool 1.0 (May 4 2005)

usage:

-m/--merge-file=<file> ...

-p/--migrate-pool=<pool> ...

-l/--list-releases

-h/--help print help

-v/--version print version info

command line reference:

-m/--merge-file=<file> merge this file with a template

-t/--template=<template> template to be used for merging

-o/--out-file=<file> write merged output to this file. (use '-' for std

output)

-p/--migrate-pool=<pool> migrate specified pool

-s/--source-release=<release> migrate from specified release [optional, deter-

mined automatically]

-r/--target-release=<release> migrate to specified release

-b/--backup create backup of all files

-d/--pools-basedir=<dir> base dir of all pools (default: /opt/myAMC/vFF)

-l/--list-releases list all available releases

misc options:

-c/--clean remove unnecessary files and config settings

-V/--verbose be verbose

-lf/--logfile <file> write to this log file

-lp/--logpath <path> create log file in this directory

FlexFrame Autonomy


Explanation of the Parameters of the FA Migration Tool

-m/--merge-file=<file> Merges the specified file with the template

-t/--template=<template> Specifies the template to be used for the

merge

-o/--out-file=<file> Write merged results into the specified file

(use '-' for standard output)

-p/--migrate-pool=<pool> Specifies the pool to be migrated

-s/--source-release=<release> Migrates from specified release (parameter

is optional). If parameter is not specified, the

version of the pool to be converted is used.

-r/--target-release=<release> Migration to the specified release

-b/--backup Generates a backup of all files

-d/--pools-basedir=<dir> Basic directory of the pools (default: /opt/myAMC/vFF)

-l/--list-releases Lists all available (installed) releases

-c/--clean Removes unnecessary files and configura-

tion settings

-V/--verbose Detailed output during migration

-lf/--logfile <file> Writes log messages to the specified log file

-lp/--logpath <path> Generates the log file in the specified direc-

tory

-h/--help Prints usage, as shown above

FlexFrame Autonomy


4.13 Command Line Interface fa_list_services.sh

The command line interface is used to list the states of all FlexFrame services.

Usage:

/opt/myAMC/scripts/fa_list_services.sh -h

usage: ./fa_list_services.sh [-ivcCH] [<pool> ...]

-i: show inactive services

-v: verbose mode

-c force use of colors for state information

-C suppress colors for state information

-H suppress headers

-h show usage

-? show usage

if no pools are specified, services of all pools will be shown.

Example:

/opt/myAMC/scripts # ./fa_list_services.sh -C p1

Pool Group Hostname SID Type Id State

p1 Trauben pw250a OEC db RUNNING

p1 g1 rx600a OEP db RUNNING

FlexFrame Autonomy


4.14 Command Execution at All Nodes of a Pool / ALL_POOLS

For execution of a command at all nodes of a pool you can use the script allnodes:

Usage:

cn2:/opt/myAMC/scripts # ./allnodes

allnodes <pool> "<command>"

.

allnodes ALL_POOLS "<command>"

allnodes ALLPOOLS "<command>"

allnodes ALL "<command>"

cn2:/opt/myAMC/scripts #

Example:

cn2:/opt/myAMC/scripts # ./allnodes FCK date

Pool: FCK

node pw250c:

Tue Jun 27 10:28:21 CEST 2006

node pw250d:

Tue Jun 27 10:28:21 CEST 2006

node pw650a:

Tue Jun 27 10:28:21 CEST 2006

node blade4:

Tue Jun 27 10:28:21 CEST 2006

node rx300a:

Tue Jun 27 10:28:21 CEST 2006

node rx300b:

Tue Jun 27 10:28:22 CEST 2006

cn2:/opt/myAMC/scripts #


5 WebInterface

The FA WebGui visualizes all nodes and services present in a FlexFrame system insofar

as they are monitored by an FA AppAgent. The display shows the status, availablity and

messages of the Application and Control Agents.

5.1 Installation / Configuration

5.1.1 Prerequisites

On the Control Node an Apache-Tomcat Servlet Container must be installed. Currently

Tomcat >= 5.5.x is supported. Additionally a Oracle Java JRE or SDK >= 1.6.0 must be

installed.

Prerequisites for the clients are FireFox >= 3.0 or Internet Explorer >= 6.0 and the Oracle

Java plugin >= 1.6.0.

5.1.2 Installation

The installation package is called myAMC.FA_WebGui-<X.Y-Z>.i386.rpm.

Install the relevant package on the Control Node with

rpm –i myAMC.FA_WebGui-<X.Y-Z>.i386.rpm

The files are installed under /opt/myAMC/FA_WebGui.

5.1.3 Configuration

5.1.3.1 Web Server

Provided no paths have been changed in the FA configuration, the configuration runs out

of the box.

Changes to the web server require Tomcat to be restarted or reloaded.

5.1.3.2 Login IDs

Login IDs are stored in the file /opt/myAMC/config/amc-users.xml. User names

and passwords can be entered simply here. No administration of rights is provided in the

current version; each user has the same rights and sees all pools, groups and systems.

WebInterface


The file’s as-supplied status is as follows:

<?xml version="1.0"?>

<users>

<user name="admin" password="FlexFrame" role="admin, user"/>

<user name="myAMC" password="FlexFrame" role="user"/>

<user name="ByD" password="FSC" role="user"/>

</users>

Each user is assigned an entry in the following format:

<user name="admin" password="FlexFrame" role="admin, user"/>

The name field contains the user name and the id field the password. The remaining

fields must be taken over as they have been defined.

The default values for user and password are:

User: myAMC

PW: FlexFrame

The configuration file is an XML file whose format must be valid, otherwise no

user can log in. You can check this by opening the file in Internet Explorer, for

example. If the file is displayed without an error message, it is also accepted by

the WebGui.

If changes are made, the FA WebGui must be restarted or reloaded (e.g. using the

Tomcat Service Manager) or Tomcat must be restarted or reloaded.

5.1.3.3 Link to the Accounting and Messenger Databases

The FA WebInterface enables messages of the Application and Control Agents to be

displayed. These are sent from the Agents to the myAMC.Messenger, which filters them,

automatically triggers reactions as required, and writes them into a database.

Additionally the FA WebGui supports showing performance graphs and accounting data,

which is read from the accounting database.

The database access parameters must be specified in the configuration file

/opt/myAMC/config/connector.properties. If changes are made, the FA

WebGui must be restarted or reloaded (e.g. using the Tomcat Service Manager) or

Tomcat must be restarted or reloaded.

Configuration of the connector properties see chapter 2.3.5.

WebInterface


5.1.3.4 LDAP Options

The graphical user interface provides structural information of pools, nodes and services

even if they are not running. They are gained by accessing the information stored in the

system-wide LDAP database.

LDAP access is preconfigured for standard installations and works out of the box. In

order to allow fine-grained control, the following options can be specified in the configura-

tion file /opt/myAMC/config/FA_WebGui.conf. If changes are made, the FA WebIn-

terface must be restarted or reloaded (e.g. using the Tomcat Service Manager) or Tomcat

must be restarted or reloaded.

The table below includes all LDAP-specific configuration settings, their default values and

a description.

LDAP database Default value and description

ldap.config.file /etc/openldap/ldap.conf

Specifies the name of the system-wide LDAP config file,

which contains parameters for all services requiring LDAP

access, e.g. automount, etc. This file will be read in order to

get the necessary settings.

ldap.username <empty>

User name for LDAP access. This parameter may contain

the BindDN (equivalent of user name).

ldap.password <empty>

Password for LDAP access.

ldap.update-cycletime 3600

Specifies how often (in seconds) structural information is

updated.

ldap.enabled true

Specifies whether LDAP access is enabled (true, false)

5.1.3.5 GUI Options

The graphical user interface provides some options which can be set using applet pa-

rameters. These permit the window size to be changed, automatic login, and the activa-

tion of the context menus.

Settings are changed using applet parameters in the

/opt/myAMC/FA_WebGui/index.html file.

Autlogin and activation of the context menus are implemented using applet parameters,

which are set as follows:

///////////////////////////////////////////////////

WebInterface


// user changable paramaters

// set to 'yes' to enable context menues

allowContextMenues:'no',

// change to enable auto-login

//user: 'MYUSER',

//password: 'MYPASSWD',

///////////////////////////////////////////////////

Parameter Possible values and description

allowContextMenues yes or true and no or false (no = default)

Specifies whether context menus (and thus actions) are per-

mitted.

user User name for Autologin; must be uncommented to be used

password Password for Autologin; must be uncommented to be used

5.1.3.6 Other Settings

Further settings can be made in the file /opt/myAMC/config/FA_WebGui.conf. If

changes are made, the FA WebInterface must be restarted or reloaded (e.g. using the

Tomcat Service Manager) or Tomcat must be restarted or reloaded.

BlackBoard settings Default value and description

The BlackBoard is an interface for transferring commands

to the Application and Control Agents (see chapter 8).

blackboard.toolpath /opt/myAMC/FA_CtrlAgent

Path containing the BlackBoard command line tool.

blackboard.toolfile BBTool.sh

Name of the BlackBoard command line tool. This is nor-

mally a shell script which starts the correct program in

accordance with the operating system .

blackboard.path /opt/myAMC/vFF/

Path for the BlackBoard file (without pool name!)

blackboard.file data/FA/blackboard/blackboard.txt

Name of the BlackBoard file (without pool name!)

blackboard.md5key myAMC.FA_BBTool

MD5 key which is used for saving.

WebInterface


Paths and file names Default value and description

path.flexframe.data /opt/myAMC/vFF

Location of the data directories (pools)

path.flexframe.config /opt/myAMC/config

Configuration directory

file.amc-roles amc-users.xml

Name of the file (in the above-mentioned configuration

directory) which contains the user definitions.

Hypervisor Monitor Default value and description

builder.enabled.vmhost true

Specifies whether Hypervisor monitoring service is ena-

bled (true or commented) or disabled.

builder.update-

cycletime.vmhost

60

Cycle time (in seconds) for Hypervisor monitor service

Reading out the FA data Default value and description

pools.update-cycletime 3

Specifies how often the data of all pools is to be read out

(in seconds).

WebInterface


5.2 Visualization

5.2.1 Starting the WebInterface / Access via Web Browser

The WebInterface is always reachable when the Apache Tomcat is running. The Apache

Tomcat will be started by the script /etc/init.d/ff_jakarta-tomcat start.

The WebInterface can be reached at the following URL:

http://<control_node>:8080/FAwebgui/

The port specified can be changed in the Tomcat configuration files server.xml.

5.2.2 Login

The user name and password must be entered in the login mask to permit authentication.

You can use the WebInterface only with a valid combination of user name and password

(for details on configuring users see section 5.1.3.2).

WebInterface


5.2.3 Overview of Elements

The WebInterface provides a clearly structured display of all the elements in a FlexFrame

system. The left-hand side shows pools, groups and nodes in a tree structure. The main

panel provides and overview of the status of all instances and displays messages from

the various agents. The panels at the lower edge show all the nodes and systems and

their current statuses.

WebInterface


5.2.4 Pool / Group Tree

5.2.4.1 Overview

The pool and group tree shows all elements of a FlexFrame system in a hierarchical

structure. Each pool has, child elements, groups, which in turn are used as containers for

systems, nodes or instances.

5.2.4.2 Status

The status of an element is indicated by its color.

The colors have the following meanings:

Red Critical

Yellow Warning

Green Normal, everything OK.

White Not active or no further information

Black Not active or shut down

Blue Managed Spare Node in Adminpool

Color inheritance takes place according to the worst-case principle. Here an element

displays the worst value for its own status or for the status of its child elements.

5.2.4.3 Selecting an Element

When an element is selected, the associated instances, nodes and systems are dis-

played in the panels on the right-hand side:

When a pool is selected, the instance view shows all instances which are running on

nodes of this pool, the node panel shows all nodes belonging to the selected pool,

and the system panel shows all systems which have instances on nodes of this pool.

When a group is selected, the instance view shows all instances which are running on

nodes of this group, the node panel shows all nodes belonging to the selected

group, and the system panel shows all systems which have instances on nodes of

this group.

When an Application Node is selected, the instance view shows all instances which are

running on this node, the node panel shows the selected node and the system panel

shows all systems which have instances on this node.

When a system is selected, the instance view shows all instances belonging to this sys-

tem, the node panel shows the nodes on which the associated instances are run-

ning, and the system panel shows the selected system.

When an instance is selected, the instance view shows the selected instance, the node

panel shows the node on which the instance is running, and the system panel shows

the system to which the instance belongs.

WebInterface


5.2.4.4 Different Tree Presentations

The tree view offers different presentations for displaying the various elements.

The Nodes tree displays the pool, group and node elements.

Example 1 Example 2

The nodes displayed in black are not currently running, the ones in white are

running, but are not currently hosting services, i.e. they are spare nodes.

Virtual nodes will be marked with a “V”.

Spare noes will be marked with a “S”.

WebInterface


The system pool “FlexFrame” shows Hypervisor, virtual machines and their state.

The state is Hypervisor-specific and does not show FlexFrame or SAP service state.

The type of Hypervisor (VMware/ESX or KVM) will be shown als overlay icon.

State: RUNNING as soon as VM is started/powered-on on Hypervisor level.

State is not related to ping-response or FA_AppAgent state

WebInterface


The Systems tree displays the pool, system and service elements.

The All tree displays all elements, from pool to service.

WebInterface


5.2.4.5 Visualization of host systems and virtual servers

As of version 9.0 the myAMC.FA WebGUI displays the hypervisor of virtual servers.

For each hypervisor the running virtual machines are shown. Virtual servers are marked

with an overlay icon “V”.

Hypervisor are shown together with the running virtual servers across the pools in the

pesudo pool "FlexFrame".

The running virtual servers of a hypervisor server can be in different FlexFrame groups

as well as in different pools.

In addition, the virtual server can be seen as any physical application nodes in their re-

spective pool.

5.2.4.5.1 Virtual server

A virtual machine is marked with an overlay icon „V“ in the tree and the node panel.

5.2.4.5.2 Hypervisor

A hypervisor is a physical server, running on the virtual node in a virtualized environment.

With FlexFrame 5.3 or FlexFrame Ochestrator V1.0 ESX and KVM hypervisors are sup-

ported.

Each Frame has a pool “FlexFrame”, which is a special system pool for information about

the whole frame. The data of the pool FlexFrame is located on the control node and not

on the storage system. The pool “FlexFrame” contains a group ”Hypervisor Nodes”, which

shows all physical servers (Hypervisor) with the currently existing virtual machines.

The shown state in pool FlexFrame is the state seen from Hypervisor / interface to Hy-

pervisor. A running VM means that from the perspective of Hypervisor the virtual machine

is started. VM state RUNNING is shown even if the operating system and application

software is not completely active.

The state of the VM with operating system and application software as seen from my-

AMC.FA-Agents is visible in normal pool. So the state of the same VM might be different

between pool FlexFrame and normal pool.

WebInterface


The following illustration shows the pool “FlexFrame” with Hypervisor and virtual servers

and the pool “pool3” with physical, virtual and sparenodes.

WebInterface


5.2.5 Status Display

5.2.5.1 Node Panel

The node panel displays all nodes on which instances belonging to the currently selected

element are running. Each node is shown with its name, icon and status, the node status

corresponding to the worst instance status.

In the node panel virtual servers are marked with an overlay icon “V”.

Nodes whose shutdown facility may not be configured right or is unknown will be shown

with a red triangle in node and tree-panel (from FlexFrame V5.2).

The order of the elements in the panel can be changed by pressing the right button and

choosing the sort order.

Example: Sorting by colours, i.e. critical nodes with problems can be recognized directly,

even if many nodes are visible in the panel.

If you leave the mouse cursor over a node icon, a tool tip is displayed specifying the

name, group and pool.

Example 1 Example 2

WebInterface


5.2.5.2 System Panel

The system panel displays all systems of the instances belonging to the current-

ly selected element. Each system is shown with its name, icon and status, the

system status corresponding to the worst instance status.

The order of the elements in the panel can be changed by pressing the right button and

choosing thesort order.

WebInterface


5.2.5.3 Instance Panel

The instance panel displays all instances belonging to the currently selected element in a

list view containing the following fields:

Severity

Current instance status: red = critical, yellow = warning, green = everything OK,

white = no information or not active.

Pool

Name of the pool incorporating the node on which the instance runs.

Group

Name of the group incorporating the node on which the instance runs.

System

Name of the system to which the instance belongs.

ID Instance number.

Service

Generic instance name.

Type

Type of instance: database (DB), central instance (CI), application instance (APP)

Priority

Priority of the instance (see section 3.3.1)

Node

Node on which the instance is running

State

Current status of the instance

WebInterface


The fields in the list can be arranged in any order by left-clicking on the column header.

Clicking on it again reverses the sort order, and a third click cancels the sort again. An

arrow appears next to the header which specifies the current sort order and sort direction.

If sorting is to take place according to several columns, you click on the first column

(twice if required, depending on whether sorting is to take place in ascending or descend-

ing order). You select all other columns with CTRL + mouse click. The lower the sort

priority of a column becomes, the smaller the arrows next to the column header become.

In order, for example, to sort by pool and within a pool by group, you first click on Pool,

hold down the CTRL key, and then click on Group.

With activate or not avcicate the button the view changes or changes not the selection in the tree at the left side.

WebInterface


5.2.5.4 Assigning States to Colors

As described above, color inheritance takes place according to the worst-case principle.

Here an element displays the worst value for its own status or for the status of its child

elements.

The states are assigned to the color display as follows:

White:

inactive or no

further information

Green:

normal,

everything ok

Yellow:

warning

Red:

critical

Black:

critical

Blue:

Info

Node states

RUNNING

SWITCH_INT

SWITCH_EXT

PowerOff

DOWN

Managed Spare

Node in Ad-

minpool

Service states

SHUTDOWN

DOWN

WATCH

UNKNOWN

NULL

RUNNING

STOPPING

STARTING

REBOOT

RESTART

RESTARTING

REBOOT

REBOOTING

RBGET

SWITCH

SWITCHOVER

SWGET

ERROR

The keywords for the states are accepted in either upper or lower case.

WebInterface


Additional of the colored visualization there are overlay icons with the following sense:

ESX Hypervisor

KVM Hypervisor

Virtual node

Spare node

Virtual spare node

Shutdown facility detect that shutdown for this node should not work (shutdown

is invalid)

NoWatch attribute of a service

These overlay icons will be shown in tree and in node panel.

WebInterface


5.2.6 Message Display

The message display provides an overview of all messages which come from the Appli-

cation and Control Agents of the FlexFrame system. The messages are prefiltered in

accordance with the element currently selected in the tree. If the highest element with the name FlexFrame is selected, all messages are displayed. If a specific node is selected,

only messages of this node are displayed.

Scrolling to reg right shown additional informative columns:

WebInterface


5.2.6.1 Fields

The following fields are available in the message panel:

Severity

Message severity: red = critical, yellow = warning, green = everything OK, white = no

information or not active

Timestamp

Message timestamp

Pool

Pool name

Group

Group name

System

System name

Id

Instance number

Type

Type of service affected (DB, APP, CI, …)

Node

Node name

ShortMessage

Abbreviated message

Category

Category

Info

Service-Type of service affected (SRV_DB, SRV_APP, SRV_CI, …)

MessageId

Unique message ID

Application

Application which sent the message

LongMessage

Detailed message

WebInterface


5.2.6.2 Navigation

The navigation bar enables you to navigate through all messages:

…

The number of messages found within the selected time range will be shown below the

messages. The example above shows, that there are 493 messages in the specified time

range.

With the message view will be closed.

Witch switch on the button the selection from the tree-panel will be adopted to the

message view. Then only the messages will be shown, which will be correlate to the

selection in tree-panel.

auto refresh automatically looks for new messages during each update cycle. With the

drop-down-menue right from “auto refresh” the refresh cycle can be adjusted.

The fields “From” and “to” specify the currently selected timerange. They can be changed

as well by manually entering a valid date. Start and end time are synchronized, i.e. the

timerange will always be the one specified below.

The buttons and can be used to scroll within the messages.

fast-forwards toward the most recent messages. The button updates the mes-

sages in the current time range. If the update will take to much time, it can be interrupted

with .

The time span specifies the amount of time by which the displayed timerange will be

adjusted into past or future, when the buttons and or next to the start and

end date are used.

A manual refresh with is necessary after the first time the message view has been

opened, and whenever start date, end date, or the time span has been changed.

Pushing a drop-down menue will be shown, where ‚page-size‘ and ‚cache‘ could be

adjusted. Using cache the message view must not be load all data from database for a

refresh. The amount of data loaded from database during refresh and will be hold in

cache, depends from ‚page size‘ and the desired timerange. If timerange is smaller as

page-size. It may be that more than one page size will be loaded, because the message

view will use page.size with special rules. If the page-size is smaller as the timerange,

there will be loaded as many page-sizes that the whole timerange is filled.

WebInterface


With the cache will be cleared.

WebInterface


5.2.6.3 Sorting

The message window allows sorting of the currently displayed messages.

The fields in the list can be arranged in any order by left-clicking on the column header.

Clicking on it again reverses the sort order, and a third click cancels sorting. An arrow

next to the header specifies the current sort order and direction.

If sorting is to take place according to several columns, click on the first column (twice if

required, depending on whether sorting is to take place in ascending or descending or-

der). Additional columns can be selected with CTRL + mouse click. The size of the arrow

next to the column header increases and decreases with sort priority.

In order, for example, to sort by group and within a group by nodes, click on “Group”, hold

down the CTRL key, and then click on “Node”.

By default messages are displayed in descending chronological order, i.e. new messages

are shown at the top.

WebInterface


5.2.7 Configuration of FlexFrame Autonomy with the Webin-terface

The FlexFrame Autonomy Agents can be configured using the web interface.

The myAMC.FA WebGui is a display option for the visualization of Flexframe infrastruc-

ture and the services running on it. The configuration with this frontend is not possible in

the standard fashion. Reason for this is that in normal operation important configuration

settings can not be changed by an operator error. An administrator can enable the con-

figuration dialogs by following change.

In order to use the configuration editor, the context menus must be enabled. This can be

achieved by setting the option allowContextMenues in the file

/opt/myAMC/FA_WebGui/index.html to yes.

Caution: Turning on the context menu to make from the pure observation GUI (sta-

tus-and event-monitor) a limited ADMIN GUI.

The configuration editor can be started by right-clicking on a pool element and selecting Edit configuration from the context menu.

The normal way to configure the myAMC.FA Suite is the myAMC.FA ConfigGui. Calling

this configuration Gui should be allowed only to authorized persons.

The following descriptions and screenshots show a matter of principle, but not the whole

configuration menus.

WebInterface


Each parameter offers a detailed description, when the mouse pointer is positioned over

the parameter name:

The values can be modified using the input fields, changes are indicated by a an icon:

All parameters are validated, whenever a change occurs. Validation failures are indicated

by an icon, and offer a detailed error description as a tooltip:

Parameter lists can be extended using the add button, unnecessary entries can be

removed using the remove button. When a new entry is created, an input dialog asks for

a name, which uniquely identifies this entry. The suggested name is composed of a

template name and the current date:

Some additional elements can be used to control the configuration process:

Save: Stores the modified values in the config files.

Revert: Reverts all changes.

Close: Closes the configuration dialog.

WebInterface


5.2.7.1 Configuration of priorities and powervalue

The high availability and autonomy rules also depend on the priorities and the configured

power requirements of a service. This parameter priority or power value to a service are

stored in the pool-specific myAMC_FA_Group.xml file.

From FlexFrame V5.3 or FlexFrame Orchestrator V1.0 there is a graphical dialog with

which these parameters can be edited in myAMC.FA ConfigGui.

An understanding of the configuration of a service associated priority value and the power

is very important because the behavior is controlled depending upon failure.

It is called in the main menu in the View menu-> Priosettings.

The configuration of the rules for priorities and power values is pool-specific. Unless a

pool is selected in the tree, a grid appears empty priorities. The DataGrid for priority con-

figuration has its own menu bar with its own menu button.

WebInterface


The buttons shown in the image above from left to right have the following meaning:

Priorities display and close the dialog

Changing the display depending on the position in the tree

Save button to save the changes.

InsertBefore A new rule is inserted before the selected rule

Insert After: A new rule by selecting the rule is placed.

Edit: A selected rule can be edited

Delete: You can delete a selected rule

Filters for display of associated services

Fine filter for selective display of services for the selected rule

Once you select a pool, the stored rules appear. Rules are shown in a different color than

the services that are derived from this rule, their priority or power value.

The picture shows the rules concerning the currently selected tree pool "default".

WebInterface


All parameters are arranged hierarchically. I.e. the rule tree is processed from top to

bottom. A service derives from its priority rule from the first hit. In the rules table is there-

fore always, and that should not be changed, as the last rule *. * Rule, which applies to

each service and defines the default priority and the default power value for a service. Is

not this a general rule, the internal default values are set for a service matched with any

rule entry.

The default values are 998 for the priority and 100000 for the Power value. If you want a

service of a particular SAP system has to be another priority is always to insert this rule

before the general rule.

By using the filter button, the currently existing services may be displayed. The services

are always displayed immediately after the rule has matched in the determination of ser-

vice priority and power value.

The display of the rules along with the to the Related Services is shown in the next pic-

ture. Together with the button 'follow selction' can be controlled so that only the services

for a group within a pool are displayed for clarity.

Insert a new rule.:

The configuration of a new rule can be made in the filtered or unfiltered display. Before

creating a new rule, the new rule must be considered where in the rules table must be

inserted. Accordingly, there is a menu button to insert a new rule before or after a select-

ed rule or to edit the selected rule.

WebInterface


The definition of such a rule is shown below:

Group: Defines the group of the pool in which the rule should apply. No entry or *

is called the rule is applied Across group within the pool.

SID: System ID to apply for a rule. No entry or * means that the rule applies to

all systems.

Service Type: The service type.

ServiceID Instancenummer a service.

The individual Attritbute generally be linked via "AND" in the evaluation. So a service is

assigned the priorities defined in the rule and power values if its attributes Group, SID,

service type, service ID match on the rule.

Evaluation is carried out hierarchically from top to bottom. The position of a rule in the

rule tree therefore has significant influence on which rule is applied to a service at run

time. For a more specific rule is applied before a general rule, this rule must be in the

table above.

The following figure shows the dialog to add a new rule. This dialog is identical in princi-

ple for 'insertBefore', 'insertAfter' and 'edit'.

So that new rules can be added to the control hierarchy at the right places, there are two

menu buttons 'insert before' and 'after insert'. Therefore it is necessary that a control is

selected.

WebInterface


Furthermore, there is still the option to edit an existing rule. A new or modified rule is

initially only in the memory of myAMC.FA ConfigGuis only on the save button, the rule

change is stored in the associated file.

Activation of a rule change:

Note: A modified rule is active after a restart of the myAMC.FA agents.

The display of the Services always displays the current values of the services as they

currently are, according to the live list file.

In the rule table, a service usually is indicating he will belong to from rule activation.

The user can therefore see the possible effects of a rule change right on the screen.

Display the rule table with and without services:

This can be set via the filter button. The filter button controls whether only the rules or the

rules derived from the service are shown.

The line filter always applies only to the selected line. The button for the row filter is only

visible when switched filter function.

WebInterface


5.2.7.2 Configuration of highavailability

The high availability and autonomy rules depend on the parameters for the control of high

availability. These parameters are configured at the pool level.

As of version V5.3 FlexFrame, in addition to the default set of parameters that applies to

the entire pool, a separate set of rules for a group within a pool can be configured.

The group-specific ruleset contains only the important SwitchOver- and TakeOver- pa-

rameter, which controls the highavailability.

The image shows the dialog for configuring the parameter settings for the Takeover

Rules.

In addition to the default configuration that applies to all groups of a pool, a rule set that

deviates can be configured for a group within a pool.

WebInterface


The configuration dialog is shown in the following image.

To add a neu group specific configuration please press „Add“.

Then the following dialog will appear:

Here you must enter the name of the group for which the new rule should apply:

WebInterface


After pressing "OK" the new group appears in the list and can be customized:

WebInterface


5.2.8 Special Signals

5.2.8.1 Overview

The pool and group tree shows all elements of a FlexFrame system in a hierarchical

structure. Each pool has, as child elements, groups, which in turn are used as containers

for systems, nodes or instances. In addition to the standard visualization of the elements

their some signaling feature for special functions or situtations.

5.2.8.2 Spare Nodes in a Spare Pool

The visualization of spare node in an admin pool uses additional signals to differentiate

between active and inacitive spare nodes. The status of an element is indicated by its

color.

A spare node in the SparePool/SpareGroup is white or black because the node is un-

known and down. To use a spare node from the spare pool in an other pool the spare

node has to be activated. In addition to the standard signaling an activated spare node in

a spare pool has an overlay icon. Spare nodes in the admin pool without overlay icon are

not activated.

WebInterface


5.3 Interaction

5.3.1 Updates

The WebInterface regularly updates all the displayed elements, states and messages.

5.3.1.1 Update Interval

The interval between two updates can be set using the context menu of the button,

which can be reached using the right mouse button:

An update interval of between three seconds and two hours can be selected.

5.3.1.2 Manual Update

You can initiate a manual update outside the update cycle using the button.

WebInterface


5.3.1.3 Reinitialization

If you wish to reinitialize and update the entire interface, you can do this using the Rei-

nitialize option in thecontext menu of the button.

5.3.1.4 Pause Mode (No Update)

If you wish to disable the update for a while (pause mode), you can do this using the

Enable Update option in thecontext menu of the button.

WebInterface


5.4 Info and Help

The button enables you to access the information and help dialogs.

Message History

The Message History displays the last messages of the WebInterface.

Status

Results of Commands like “Start Service…” etc.

WebInterface


5.5 FlexFrame Performance and Accounting Plug-in

The FlexFrame Autonomous Management Center contains, as optional components,

plug-ins for the views of the performance and accounting data.

The plug-in consists of a graphical display in which the performance or accounting values

are shown in the form of graphs. The temporal view domain can be freely chosen and is

only limited through the data timespans stored in the Repository.

The GUI uses a powerful cache algorithm to optimize the display, which minimizes the

waste of resources and optimizes the response time for the data request. The cache

works through page size and number of pages, meaning that, according to the efficiency

of the front-end system used, the default parameters can be temporarily altered to adjust

the signal performance to individual demands.

Pagesize

Pagecount

These two parameters influence the size of the display cache and the quantity of data to

be transferred at the request of the GUI to its server.

The cache algorithm erases old pages only after obtaining the page count. Then the

pages not used for the longest time are erased.

The performance plug-in shows the CPU and memory values for a chosen service, in

each case in individual graphs. Every graph shows minimum, average and maximum

values.

The displayed timespan can be shifted using the two arrows at the top of the plug-in.

5.6 FlexFrame Reporting Plug-in

The FlexFrame Reporting plug-in contains the production, export and print of reports from

the FlexFrame Autonomous Management Center. This plug-in is only available in the

standalone variant of the management center; display within a web browser is not possi-

ble.

The plug-in shows a selection of available reports. New predefined or user-specified

reports can simply be filed in the report directory and are immediately applicable.

All reports have a standardized parameter interface. A new report can easily be derived

from one of the existing reports.

WebInterface


5.7 Customization of Icons in the FA WebGUI

The FA WebGUI uses icons to represent company and product information as well as

nodes and systems. As of version 8.0 of the FA WebGUI, these icons can be changed by

the user.

User-defined icons can be deposited in directory /opt/myAMC/FA_WebGui/icons. The

file /opt/myAMC/FA_WebGui/FlexFrameViewer.properties contains names of

heading, logo, node, and system icons as well as the window title and can be customized

as necessary.

The information stored in this directory (logos and icons) are loaded each time the FA

WebGUI, and then stored locally for the duration of the use of the FA WebGUI. I.e. the

start of the FA WebGUI will ever slow down on the number of individualized icons and

logos, due to the additional load times slightly.

The behavior and display of the FA WebGUI remain unchanged if the directory for the

logos and icons contains no additional logos and icons. The directory and the standard

logos and icons are already components of the distributed FA WebGUI versions. The

exact procedure is described in section 7.5.14.


6 FlexFrame Autonomy Power Shut-down Concept

6.1 General

The FlexFrame Autonomy power shutdown function is designed to provide a simple,

easy-to-configure method for implementing secure shutdown of various hardware plat-

forms.

For this pupose the FlexFrame Autonomy Control Agents use the generic interface with a

set of predefined shutdown methods.

A new parameter defines the check sequence to find the appropriate shutdown method

for a server.

For external SwitchOver of application instances, FlexFrame Autonomy requires a secure

option for shutting-down nodes simply and securely.

The FlexFrame Autonomy Agents ascertain this lack of configuration information at

runtime and then generate the configuration information required for the agents.

FA_Blade

FA_IPMI

FA_VM

FA_HV

FA_Generic

Reserved

FlexFrame Autonomy Power Shutdown Concept


6.2 Power Shutdown Architecture

The figure below provides an overview of the components involved and how these inter-

work in a FlexFrame environment.

The FlexFrame Autonomy Control Agent runs on the active Control Node. The FlexFrame

Autonomy Application Agents provide the Control Agent with information on the computer

type and further information which is required for generic detection of shutdown infor-

mation, insofar as this is technically possible and the information is unambiguous and can

be ascertained securely. Information which cannot be ascertained generically must be

entered manually in the Powershutdown section of the myAMC_FA.xml file.

vFF Pool ...n

Data

Config Logfiles

vFF Pool2



6.3 Basics

6.3.1 Powershutdown methods for use with FA generic

shutdown facility

As of FlexFrame version 4.2B, the generic myAMC.FA Powershutdown facility uses

wrapper scripts to integrate server specific shutdown tools.

The standard path for the myAMC.FA shutdown wrapper scripts is:

/opt/myAMC/scripts/PowerMng/<type>_wrapper.sh

6.3.1.1 Power Shutdown for Blade Systems

Power shutdown for the Blade systems is implemented via the wrapper script blade_wrapper.sh. In order to shutdown a compute blade, the IP addresses, SNMP

communities and password (usually the same as the SNMP community) of the Manage-

ment Blade must be configured.

The hosting blade chassis, MAC addresses and slot numbers of compute blades are

determined automatically.

The wrapper script is: blade_wrapper.sh

6.3.1.2 Power Shutdown for PRIMERGY Systems

By default power shutdown for the PRIMERGY systems in FlexFrame is performed via

the IPMI interface using the Linux IPMI tool. Necessary configuration settings are user

name and password, if they are different from the default user name and password.

The wrapper scipt is: ipmi_wrapper.sh



6.3.1.3 Power Shutdown for virtual VM Server

Power shutdown for virtual servers is performed via the VM shutdown script.

The wrapper scripts are: vm_vm_wrapper.sh and vm_hv_wrapper.sh

6.3.1.4 Power Shutdown for other Server (e.g. PrimeQuest PQ2800

Power shutdown for other servers like PrimeQuest PQ2800 is performed via the generic

shutdown script.

The wrapper script is: generic_wrapper.sh

6.3.1.5 Power Shutdown for Host systems of Virtual Servers

The usage of virtual servers on a host system (ESX server) requires the configuration of

the shutdown interface of the host system. The host system can be a blade system or a

rack server, therefore there are different shutdown parameter.

The shutdown parameter of the host systems have to be configured manually, since the

myAMC.FA-App agents cannot provide information about these hostsytems.

The following values must be known for the hostsystem (ESX server):

Hostname: hostname of the host system

If the hostsystem is a bladesystem, additional information is necessary:

MAC address: mac address of the host system

The requirement is that the authentication defaults and the authentication settings (SNMP

community, user name, password) and the management blade are configured correctly.



6.4 Power Shutdown Configuration

The following sections describe the configuration of the power shutdown. This can also

be done in the myAMC Webinterface.

See Pool-Konfiguration „Shutdown Settings“

For general information about usage and configuring with the myAMC Webinterface

please see section 5.



6.4.1 Switchover Control Parameters

The normal lapse on breakdown of a node requires the disconnecting of this node to

avoid a disturbance caused by the existence of duplicate services. If it is not possible to

disconnect the node, the move breaks up with a failure notice. The services concerned

are not restarted.

With the help of the parameter IgnoreShutdownFailure, this failure can be influenced

in that, if a shutdown fault occurs, for example because a shutdown parameter was not

configured for a node, the relevant node is separated from the network.

If a node is separated from the network, the network must be reactivated manually. For

this, the following command must be executed on the Control Node before the defective

node is restarted:

ff_sw_ports.pl --op up --name <hostname>

If the network interfaces of the node are not reactivated, it’s not possible to start the node.

See Pool-Konfiguration „Shutdown Settings“ „Shutdown Parameters “.

Description:



• Default ShutdownMode

Default mode for Shutdown (leave-off / cycle).

• Cycletime

Cycletime (ms) of Shutdown-Facility.

• Execution timeout

Timeout (ms) for the Shutdown-Agents.

• Poweroff try count

Maximum number of PowerOff attempts.

• Renew config cycletime when invalid

Cycletime (ms) to renew invalid Shutdown-configuration.

• Renew config cycletime when valid

Cycletime (ms) to renew valid Shutdown-configuration.

• Resend trap if shutdown valitity is valid

Resend time (s) of the traps for valid Shutdown-configuration.

• Resend trap if shutdown valitity is invalid

Resend time (s) of the traps for invalid Shutdown-configuration.

• SNMP timeout

Timeout (ms) for SNMP-Requests.

• SNMP try count

Number of attempts for SNNMP requests.

• Ignore Shutdown failure

Specifies if a shutdown error should be igoriert.

• PrePoffHookPath

Path to PrePoffHook-Script.

• PostPoffHookPath

Path to PostPoffHook-Script.

• Ignore PrePoffHook failure

Specifies if a PrePoffHook error should be igoriert.

• VMHostPoffCheckPath

Path to VMHostPoffCheck-Script.

• VMHostPoffCheck timeout

Timeout (s) of VMHostPoffCheck-Script.



6.4.2 User, Password and Community

To use agent power shutdown, user, password and community must be defined in the

configuration of the FA Agent. This configuration is specified in the pool-specific configu-

ration file.

The entries for user and password must be the same as those configured for IPMI (rack

server).

The entries for the community must be the same as those configured in the Management

Blade (blade server).

Further information on the configuration of user, password and community in the Applica-

tion Nodes can be found in the ‘Hardware Characteristics Quickguides for the rack serv-

ers resp. management blades’.

The configuration of default user, password and community can be found in the file my-

AMC_FA_SD_Sec.xml at config section Security_default.

In addition, individual Application Nodes can be defined with other values for user, pass-

word and community. To permit this, a corresponding configuration section must be de-

fined for the Application Node.

See Pool-Konfiguration „Shutdown Settings“ „Authentification Defaults“.

Description:

• SNMP Community

Default SNMP community for SNMP requests.

• User name

Default user name used for Shutdown.

• Password

Default password used for Shutdown.



See Pool-Konfiguration „Shutdown Settings“ „Authentification Settings“.

Description:

With "Add", a new list item is added.

With "Remove", the selected list entry will be removed.

• Name

Name of ths list entry.

• Hostname

Hostname of the host for this ‘Authentication Settings’.

• SNMP Community

SNMP community for SNMP requests.

• User name

User name used for Shutdown.

• Password

Password used for Shutdown.

Example file myAMC_FA_SD_Sec.xml:



<configsection name="Security_default">

<configentry name="Default_User">

<value type="String">OEM</value>

</configentry>

<configentry name="Default_Passwort">

<value type="String">password/value>

</configentry>

<configentry name="Default_Community">

<value type="String">public</value>

</configentry>

</configsection>



<configsection name="Sec_pq28">

<configentry name="Sec_Hostname">

<value type="String">pq28</value>

</configentry>

<configentry name="Sec_User">

<value type="String">Administrator</value>

</configentry>

<configentry name="Sec_Passwort">

<value type="String">passwort</value>

</configentry>

<configentry name="Sec_Community">


</configentry>

</configsection>



<configsection name="Sec_rx600s6">

<configentry name="Sec_Hostname">

<value type="String">rx600s6</value>

</configentry>

<configentry name="Sec_User">

<value type="String">OEM</value>

</configentry>

<configentry name="Sec_Passwort">

<value type="String">password</value>

</configentry>

<configentry name="Sec_Community">


</configentry>

</configsection>



6.4.3 Management Blades

As the Management Blades cannot be detected by the FA CtrlAgent automatically, they

must be configured. This is done in the Managementblades configuration section of the

myAMC_FA.xml configuration file. Further the SNMP community should be configured.

The example below illustrates the configuration of the Management Blade BX900-

control. The name of the configuration section (Mgmt_Blade_1) is arbitrary. A condi-

tion is that this configuration section must be contained within the Managementblades

configuration section. Further Management Blades can be entered using the relevant

configuration sections (see the otherMgmtBlade example).

See Pool-Konfiguration „Shutdown Settings“ „Management blades“.

Description:



• Name


• Hostname

Hostname (-co interface) of the management blade of the blade-chassis.

Example from File myAMC_FA.xml:



6.4.4 Application Nodes

Normally the shutdown procedures for all Application Nodes are determined automatical-

ly. However, it may be necessary to configure individual systems for power management

yourself. To permit this, the values required must be entered in the pool-specific configu-

ration file.

See Pool-Konfiguration „Shutdown Settings“ „Individual Nodes“.

Description:



• Name


• Hostname

Hostname of the host with indifidual settings.

• IP Address

IP address of the host.

• MAC Address

MAC address of the host.

No ‘:’, to mac’s separated by blank (aabbcc112233 aabbcc112233)

• Machine

Machine (empty).

• Port

Port (empty).

• ShutdownMode

Mode for Shutdown (leave-off / cycle).

• Shutdown typ

Type of Shutdown (FA_BLADE, FA_IPMI, FA_GENERIC, …).



• Hardware type

Hardware type (LINUX, UNKNOWN).

• Console

Console (empty).

Which values must be specified for which power shutdown type is shown in the table

below.

In this table there are some shutdown types resp. server types which are not supported in

the actual FlexFrame Release. For compatibility, they are included in FA-Agents and will

be documented here.

Yes Must be specified.

No Can be left empty. configentry is mandatory, however!

Default (Def) Is read from the Security_default or

Default_ShutdownMode configuration section if it is empty. If a value

is entered, it overrides the default value.

Table see next page:

(for the up-to-date types of server models see FlexFrame Support-Matrix)



Linux Linux Linux Linux ESX ESX

Value BLADE IPMI VM BLADE IPMI

Server

type

BX600

BX900

RX300Sn

RX600Sn

PQ2800B ESXVM

BX600

BX900

RX300Sn

RX600Sn

Host

name Yes Yes Yes Yes Yes Yes

Shutdown

type FA_BLADE FA_IPMI FA_GENERIC FA_VM FA_BLADE FA_IPMI

MAC

address Yes No No No Yes No

Manage-

ment

Blade IP

No No No No No No

Hardware No No No No No No

Shutdown

mode Yes/Def. Yes/Def. Yes/Def.

Yes/Def

. Yes/Def. Yes/Def.

IP ad-

dress

(Control

LAN)

No No No No No No

Console No No No No No No

Machine No No No No No No

Port No No No No No No

SNMP

communi-

ty

Yes/Def. No No Yes/Def

. Yes/Def. No

User No Yes/Def. Yes/Def. Yes/Def

. Yes/Def. Yes/Def.

Password No Yes/Def. Yes/Def. Yes/Def

. Yes/Def. Yes/Def.



6.4.4.1 Configuration of Hypervisor

The shutdown methods are usually found out automatically for all application nodes. On

the Application Nodes there is running an FA-App agent, which determines some values

required for shutdown facility (e.g. MAC-Addr of a blade).

Hint:

MAC address (2 macs separated by blank) Format: aabbcc112233 aabbcc112233

No AppAgent runs on an Hypervisor and the automatic determination of some values

can’t be done.

For this reason the Hypervisor must be configured manually.

Prerequisite for the configuration is, that the "Authentication Defauls" and "Authentication

setting" (SNMP-Community, user, PW) as well as the "management blades" are config-

ured correctly.

The configuration of the Hypervisor will be done in the pool configuration.

See Pool-Konfiguration „Shutdown Settings“ „Individual Nodes“.

6.4.4.2 Example for configuration of a node from file myAMC_FA.xml



<configsection name="ddc1esx1">


<value type="String">ddc1esx1</value>

</configentry>

<configentry name="ShutdownTyp">

<value type="String">FA_BLADE</value>

</configentry>

<configentry name="MacAddress">

<value type="String">0016363fcc08 0016363fcc0a</value>

</configentry>

<configentry name="Hardware">

<value type="String">LINUX</value>

</configentry>

<configentry name="ShutdownMode">

<value type="String">leave-off</value>

</configentry>

<configentry name="IP_Address">

<value type="String"></value>

</configentry>

<configentry name="Slot">


</configentry>

<configentry name="MgmtBladeIp">


</configentry>

<configentry name="Console">


</configentry>

<configentry name="Machine">


</configentry>

<configentry name="Port">

<value type="UnsignedInteger"></value>

</configentry>

</configsection>



6.4.5 Default Shutdown Mode

A default shutdown mode is defined for the power shutdown. This can be changed by the

configuration entry ShutdownMode.

The possible values are cycle or leave-off.

See Pool-Konfiguration „Shutdown Settings“ „Shutdown Parameters“.--> “De-

fault Shutdown mode”.

You are advised not to change the default shutdown mode so as to guarantee that Appli-

cation Nodes are shut down securely.

<configentry name="Default_ShutdownMode">


</configentry>

6.4.6 Management blades with SNMP-community different from default

For management bades with a SNMP community different from default the authentica-

tionn settings ‚comunity‘, ‚user‘, ‚pasword‘ must be explicitly configured at „Authentica-

tion Settings“.

This is independent from the discosure of the management blade in menu „Management

Blades“.



6.5 Possible Errormessages with old Configuration

The shutdown methods which are not supported in FF5.1A are removed from configura-

tion.

When running with old config you can ignore hourly messages in

/opt/local/myAMC/vFF/vFF_<pool>/log/CtrlAgt/log_user_cn1.log like:

2012-02-27 07:48:16; 1330325296; KEY_CRITICAL ; cn1-a; LogEx-

tern(); LogEx-Severity = '13' ==> log_key = '1', Fkt = 'CSHUT-

Down::WriteIpmiCfg', text = 'file /etc/opt/SMAW/SMAWsf/SA_ipmi.cfg

can not been opened for writing'.

These messages may be ignored and have no effect to the functionality.

If the customer wants to remove the old shutdown methods from existing pools there may

be changes in the pool specific configuration as follows:

To obtain the changes for existing pools in GUI, the file 'config-specs.xml' must

be copied from template-config to pool-config.

See in template:

/opt/myAMC/vFF/Common/.vFF_template.V90A00K21/config/config-specs.xml

The whole configsection ‘ <configsection name="Shutdown_Facility">’ in file

myAMC_FA.xml may be changed to that from template.

See in template:

/opt/myAMC/vFF/Common/.vFF_template.V90A00K21/config/myAMC_FA.xml

part ‘ <configsection name="Shutdown_Facility">’.

The Versionstring in path of template („V90A00K21“) is only an example. Just use the

newest version.


7 Parameter Reference

7.1 FA Agents

Operation of the FA Agents does not necessarily require individual parameterization.

After installation executable parameter files are available. For productive use, the values

must have been tested and, if necessary, adjusted to the requirements and the start,

stop, ping and restart times called for in the customer system for the services monitored

in myAMC.FA.

The monitoring of additional services which are not included in the standard rules is per-

formed in the rule file myAMC.FA_Rules.xml.

The default parameters of the FA Agents are set such that, when the performance and

accounting option is activated, no changes to the parameter settings are necessary.

7.1.1 FA Agent Configuration Files

The configuration of FlexFrame Autonomy is stored in several configuration files that can be found in /opt/myAMC/vFF/vFF_<poolname>/config. The files are in XML format.

TrapTargets.xml

Trap targets (pool-specific). The recipients of SNMP trap messages can be defined.

myAMC_FA.xml

FA Autonomy (pool-specific). Settings for autonomous reactions.

myAMC_FA_Checks.xml

Configuration of SpaceUtil-functionality..

myAMC_FA_Groups.xml

Groups (pool-specific). The group memberships are configured.

It is used as LDAP cache.

myAMC_FA_Hosts.xml

Hosts (pool-specific). The host configuration is configured.

Generated from script ‚fa_createHostFileFromLDAP.sh’.

It is used as LDAP cache.

myAMC_FA_Rules.xml

Generic service configuration

myAMC_FA_SD_Sec.xml

FA Shutdown Security (pool-specific). Settings for the Power-Shutdown.

myAMC_FA_SelectionRules.xml

Definition of the „SelectionRules“ for „Pool independent spare node“.

Parameter Reference


For most configuration files, there is an additional default file named

<filename>-default.xml which contains the as-supplied status is available after

installation. The myAMC_FA_default.xml file is an exact copy of myAMC_FA.xml in its

original state.

7.2 SNMP Traps

7.2.1 General

The TrapTargets.xml file contains all the information which is needed to send SNMP

traps. Two parameters are required for each target:

Host name or IP address

SNMP community

The community corresponds roughly to a password. Mostly the default value public

suffices. For FlexFrame at least the two Control Nodes have to be set as trap targets.

7.2.2 Structure

The file is in XML format and has the following structure:

<?xml version="1.0" encoding="ISO-8859-1"?>

<configuration>

<configsection name="SnmpConnector">

<configsection name="TrapSender">

<configsection name="TrapTargets">





<configsection name="control1">

<!—- host name of ip address or name of host to send SNMP trap to -->

<configentry name="Host">

<value type="String">control1</value>

</configentry>



<configentry name="Community">


</configentry>

</configsection>

</configsection>

</configsection>

</configsection>

Parameter Reference


</configuration>

The name or IP address of the node which is to receive the traps is specified as the value of the parameter Host (<configentry name="Host">), the SNMP community as the

value of the parameter Community (<configentry name="Community">).

A trap can be sent to any number of targets. If further targets are to be defined, the en-

closing ConfigSection must be copied, renamed and adapted accordingly.

Example:

[header, see above]







</configentry>




</configentry>

</configsection>







</configentry>




</configentry>

</configsection>



[footer, see above]

Parameter Reference


7.2.3 Default Parameter File


<configuration>

<configsection name="SnmpConnector">

<configsection name="TrapSender">

<configsection name="TrapTargets">








</configentry>




</configentry>

</configsection>







</configentry>




</configentry>

</configsection>



</configsection>

</configsection>

</configsection>

</configuration>

Parameter Reference


7.3 Pooling and Grouping

7.3.1 Pooling

Pool creation means the flexible assignment of Application Nodes to a pool. The Flex-

Frame Autonomy Agents take over the pool information from the FlexFrame configuration

in the LDAP.

When an Application Node is restarted, a start script for the FA Agents is called. This

start script determines the pool to which this Application Node belongs and starts the

FA_AppAgent accordingly.

7.3.2 Grouping

FlexFrame offers grouping functions for flexible server farming. Grouping enables nodes

and services within a pool to be assigned to different groups. A group is therefore always

a part of a virtual FlexFrame pool.

FlexFrame Autonomy grouping is a function which is configured via LDAP. Individual

parameterization is not required for the Autonomy Agents.

The Groups configuration will be read from LDAP during start of FA-Agents and be stored

in the myAMC_FA_Groups.xml file. It is used as LDAP cache.

An absolute group configuration can be implemented in this file, as in the LDAP configu-

ration. However, generic parameterization is also possible as an alternative. This can

also be used if absolute grouping is not to be used via the LDAP.

7.3.2.1 LDAP Grouping

LDAP grouping is performed in conjunction with the general configuration of the nodes in

the LDAP directory. Here the FlexFrame Autonomy Agents take over the group infor-

mation from the LDAP directory at startup, as they do the pool information. Individual

group configuration in the myAMC_FA_Groups.xml file is thus no longer required.

7.3.2.1.1 Manual Group Assignment

The group assignment is entered manually in the configuration file.

7.3.2.1.2 Example

<gr:group schema="default" name="GR_All">

<gr:description></gr:description>

<match category="${node-hostname}">

<value>vade*</value>

<value>yod*</value>

<value>blade_a*</value>

<value>blade_b*</value>

Parameter Reference


<value>server*</value>

</match>

</gr:group>

<gr:group schema="default_OS" name="GR_Solaris">




<value>blade_sol*</value>


</match>

</gr:group>

7.3.2.2 Generic Grouping

Generic group creation is implemented on the basis of generic information which the

Application Agents can obtain automatically. For generic group creation it makes sense to

use the host names, the IP addresses or the operating system employed.

The group name is also created generically. For this purpose each schema is assigned a

group naming rule which combines a fixed component with a variable component.

The following base values are available for creating the generic rules:

Operating system (os)

Network (network)

Number of CPUs (cpu)

The following generic group creation rules result:

Group definition by operating system, network and number of CPUs schema="os_network_cpu"

name="AutoGroup_${OS-typ}_${CPU-cnt}cpu_${ip-adr:netmask=/24}"

Group definition by operating system and number of CPUs schema="os_cpu"

name="AutoGroup_${OS-typ}_${CPU-cnt}cpu"

Group definition by operating system and network schema="os_network"

name="AutoGroup_${OS-typ}_${ip-adr:netmask=/24}"

Group definition by network and number of CPUs schema="network_cpu"

name="AutoGroup_${CPU-cnt}cpu_${ip-adr:netmask=/24}"

Group definition by operating system schema="os"

name="AutoGroup_${OS-typ}"

Parameter Reference


Group definition by network schema="network"

name="AutoGroup_${ip-adr:netmask=/24}"

Group definition by number of CPUs schema="cpu"

name="AutoGroup_${CPU-cnt}cpu"

The parameters are combined via a group schema. The parameters for multiple group

schemas can thus be stored in one myAMC_FA_Groups.xml file. A schema is activated

in myAMC_FA.xml.

7.3.2.2.1 Logical operation

The following logical operations are possible:

OR (through multiple value lines within a match condition)



<value>yod*</value>




</match>

AND (through multiple match conditions within a group definition)

<match category="${node-powervalue}">

<value>1000</value>

</match>





</match>

7.3.2.2.2 Wildcards

The following wildcards can be used:

* “*“ is equivalent to any number of arbitrary characters.

? “?“is equivalent to one arbitrary character.

7.3.2.2.3 Min / Max values

For numeric variables, the following syntax can be used to inquire value ranges:

syntax="min" (in this case a variable is used as a minimum)

Parameter Reference


<match category="${CPU-cnt}" syntax="min">

<value>4</value>

</match>

syntax="max" (in this case a variable is used as a maximum)

<match category="${ip-adr}" syntax="max">

<value>192.168.10.21</value>

</match>

7.3.2.2.4 Masking with IP addresses

With IP addresses, a type of masking similar to the netmask can be used in order to se-

lect network segments:

${ip-adr:netmask=/24} (in this case only the first 24 bits of the IP address are tak-

en into account)

<match category="${ip-adr:netmask=/24}">

<value>192.168.10.0</value>

</match>

7.3.2.2.5 Example



<!—

DO NOT EDIT !!!

If you need your own group definition, please use the section

before !!!

-->

<gr:group schema="os_network_cpu" name="AutoGroup_${OS-typ}_

${CPU-cnt}cpu_${ip-adr:netmask=/24}">

<gr:description>Group defined by number of CPUs, network and

operating system</gr:description>

</gr:group>

<gr:group schema="os_cpu" name="AutoGroup_${OS-typ}_${CPU-cnt}cpu">

<gr:description>Group defined by number of CPUs and operating

system</gr:description>

</gr:group>

<gr:group schema="os_network" name="AutoGroup_${OS-typ}_

${ip-adr:netmask=/24}">

<gr:description>Group defined by number of CPUs and

network</gr:description>

</gr:group>

<gr:group schema="network_cpu" name="AutoGroup_${CPU-cnt}cpu_

${ip-adr:netmask=/24}">

<gr:description>Group defined by number of CPUs</gr:description>

Parameter Reference


</gr:group>

<gr:group schema="os" name="AutoGroup_${OS-typ}">

<gr:description>Group defined by operating system</gr:description>

</gr:group>

<gr:group schema="network" name="AutoGroup_${ip-adr:netmask=/24}">

<gr:description>Group defined by network</gr:description>

</gr:group>

<gr:group schema="cpu" name="AutoGroup_${CPU-cnt}cpu">

<gr:description>Group defined by number of CPUs</gr:description>

</gr:group>

Parameter Reference



Hint:

As default there is a "service-powervalue" of 100000 for some services (DBORA,

DBSAP, DB2, CI, JC, J, LC, APP, and unknown Service-Types) for safety.

A DB (and services-types above) should not be added with add-rule to a node with other

services, cause it may have too much load.

This may be changed for customer needs.




<defs:definitions

xmlns:defs="myAMC/Definitions/1.0"

xmlns:gr="myAMC/Groups/1.0"

xmlns:attr="myAMC/Attribute/1.0">

<gr:services>





<gr:service schema="default_trivial" name="Default">



<attr:attribute name="service-priority" value="1"/>

<attr:attribute name="service-powervalue" value="2201"/>

</gr:service>





<gr:service schema="default" name="Default">

<match category="${service-type}">

<value>SRV_DBORA</value>

</match>




</gr:service>

Parameter Reference




<value>SRV_SAPSYBASE</value>

</match>




</gr:service>



<value>SRV_DBSAP</value>

</match>




</gr:service>



<value>SRV_DB2</value>

</match>




</gr:service>



<value>SRV_SCS</value>

</match>




</gr:service>



<value>SRV_ASCS</value>

</match>




</gr:service>



<value>SRV_CI</value>

</match>




</gr:service>



<value>SRV_JC</value>

</match>




</gr:service>

Parameter Reference




<value>SRV_J</value>

</match>




</gr:service>



<value>SRV_LC</value>

</match>




</gr:service>



<value>SRV_APP</value>

</match>




</gr:service>



<value>SRV_ERS</value>

</match>




</gr:service>



<value>SRV_MDS71</value>

</match>




</gr:service>



<value>SRV_MDIS71</value>

</match>




</gr:service>



<value>SRV_MDSS71</value>

</match>




Parameter Reference


</gr:service>



<value>SRV_XPR</value>

</match>




</gr:service>



<value>SRV_CMS</value>

</match>




</gr:service>



<value>SRV_TRX</value>

</match>




</gr:service>



<value>SRV_BOE4</value>

</match>




</gr:service>



<value>SRV_BOE_SRV</value>

</match>




</gr:service>



<value>SRV_SMD</value>

</match>




</gr:service>



<value>SRV_PAI</value>

</match>



Parameter Reference



</gr:service>



<value>SRV_VMWARE</value>

</match>




</gr:service>



<value>SRV_ORACLE_RAC_CRS</value>

</match>




</gr:service>



<value>SRV_ORACLE_RAC_DB</value>

</match>




</gr:service>



<value>SRV_CTM</value>

</match>




</gr:service>



<value>SRV_WEBDISP</value>

</match>




</gr:service>



<value>SRV_SMD73</value>

</match>




</gr:service>



<value>SRV_HANA</value>

</match>


Parameter Reference




</gr:service>



<value>*</value>

</match>

<match category="${system-id}">

<value>*</value>

</match>




</gr:service>





<gr:service schema="Example_service_type" name="Service_${service-

type}">



</match>




</gr:service>


type}">



</match>




</gr:service>


type}">



</match>




</gr:service>



<gr:service schema="Example_service_type" name="other">


<value>*</value>

</match>


<value>*</value>

</match>

Parameter Reference





</gr:service>





<gr:service schema="Example_static" name="ProdDB">



<value>P*</value>

</match>



<value>SRV_SAPSYBASE</value>

<value>SRV_DBSAP</value>

<value>SRV_DB2</value>

</match>




</gr:service>





<gr:service schema="Example_Customer" name="Example_Customer">



</match>


<value>PRE</value>

</match>

<match category="${service-id}">

<value>24</value>

</match>




</gr:service>




</match>


<value>PRE</value>

</match>


<value>26</value>

</match>




</gr:service>

Parameter Reference


<value>SRV_DB*</value>

</match>


<value>PPI</value>

</match>




</gr:service>




</match>


<value>PPI</value>

</match>




</gr:service>




</match>


<value>PPI</value>

</match>




</gr:service>




</match>


<value>PRE</value>

</match>


<value>27</value>

</match>




</gr:service>




</match>


<value>PBW</value>

</match>




</gr:service>


Parameter Reference




</match>


<value>PBW</value>

</match>




</gr:service>




</match>


<value>PBW</value>

</match>




</gr:service>




</match>


<value>PRE</value>

</match>




</gr:service>




</match>


<value>PRE</value>

</match>




</gr:service>




</match>


<value>P10</value>

</match>




</gr:service>




Parameter Reference


</match>


<value>P10</value>

</match>




</gr:service>




</match>


<value>PSM</value>

</match>




</gr:service>




</match>


<value>PSM</value>

</match>




</gr:service>




</match>


<value>PSM</value>

</match>




</gr:service>




</match>


<value>PBW</value>

</match>


<value>28</value>

</match>




</gr:service>



Parameter Reference



</match>


<value>QRE</value>

</match>




</gr:service>




</match>


<value>QRE</value>

</match>




</gr:service>




</match>


<value>TRE</value>

</match>




</gr:service>




</match>


<value>TRE</value>

</match>




</gr:service>




</match>


<value>Q10</value>

</match>




</gr:service>




</match>

Parameter Reference



<value>Q10</value>

</match>




</gr:service>




</match>


<value>T10</value>

</match>




</gr:service>




</match>


<value>T10</value>

</match>




</gr:service>




</match>


<value>QBW</value>

</match>




</gr:service>




</match>


<value>QBW</value>

</match>




</gr:service>




</match>


<value>QBW</value>

Parameter Reference


</match>




</gr:service>




</match>


<value>TBW</value>

</match>




</gr:service>




</match>


<value>TBW</value>

</match>




</gr:service>




</match>


<value>TBW</value>

</match>




</gr:service>




</match>


<value>QPI</value>

</match>




</gr:service>




</match>


<value>QPI</value>

</match>


Parameter Reference




</gr:service>




</match>


<value>QPI</value>

</match>




</gr:service>




</match>


<value>TPI</value>

</match>




</gr:service>




</match>


<value>TPI</value>

</match>




</gr:service>




</match>


<value>TPI</value>

</match>




</gr:service>



<value>*</value>

</match>


<value>*</value>

</match>




Parameter Reference


</gr:service>



</gr:services>







<gr:groups>









<gr:group schema="default" name="GR_All">




<value>yod*</value>




</match>

</gr:group>

<gr:group schema="default_LDAP" name="LD_Solaris_8">

Parameter Reference



<match category="${OS-version}">

<value>5.8</value>

</match>

</gr:group>

<gr:group schema="default_LDAP" name="LD_SLES_8">



<value>SuSE SLES-8*</value>

</match>

</gr:group>

<gr:group schema="default_LDAP" name="LD_Solaris_9">



<value>5.9</value>

</match>

</gr:group>

<gr:group schema="default_LDAP" name="LD_SLES_9">



<value>*SUSE LINUX Enterprise Server 9*</value>

</match>

</gr:group>

<gr:group schema="default_OS" name="Solaris_8">



<value>5.8</value>

</match>

</gr:group>

<gr:group schema="default_OS" name="SLES_8">



<value>SuSE SLES-8*</value>

</match>

</gr:group>

<gr:group schema="default_OS" name="Solaris_9">



<value>5.9</value>

</match>

</gr:group>

<gr:group schema="default_OS" name="SLES_9">



<value>*SUSE LINUX Enterprise Server 9*</value>

</match>

</gr:group>

<gr:group schema="BSP_OStyp_1" name="BSP_GR_default_${OS-typ}">


<match category="${OS-typ}">

Parameter Reference


<value>${OS-typ}</value>

</match>

<match category="${CPU-cnt-phys}" syntax="min">

<value>0</value>

</match>

<match category="${mem-total(MB)}" syntax="min">

<value>20</value>

</match>


<value>158.92.4.0</value>

</match>

<match category="${node-powervalue}" syntax="min">

<value>0</value>

</match>






</match>

</gr:group>

<gr:group schema="BSP_static_1" name="BSP_MyCompany_${OS-typ}">


<match category="${OS-typ}">

<value>${OS-typ}</value>

</match>

<match category="${CPU-cnt-phys}" syntax="min">

<value>4</value>

</match>

<match category="${mem-total(MB)}" syntax="min">

<value>4000</value>

</match>


<value>192.168.10.0</value>

</match>


<value>1000</value>

</match>





</match>

</gr:group>

<gr:group schema="BSP_static_2" name="BSP_MyCompany">


<match category="${CPU-cnt-phys}" syntax="max">

<value>2</value>

</match>


<value>192.168.10.0</value>

</match>


<value>1000</value>

</match>


Parameter Reference





</match>

</gr:group>

<gr:group schema="BSP_static_3" name="BSP_3">


<match category="${ip-adr}" syntax="min">

<value>192.168.10.17</value>

</match>

<match category="${ip-adr}" syntax="max">

<value>192.168.10.21</value>

</match>


<value>1000</value>

</match>


<value>vad*</value>




</match>

</gr:group>









<gr:group schema="os_network_cpu" name="AutoGroup_${OS-typ}_${CPU-

cnt-phys}cpu_${ip-adr:netmask=/24}">

<gr:description>Group defined by number of CPUs, network and


</gr:group>

<gr:group schema="os_cpu" name="AutoGroup_${OS-typ}_${CPU-cnt-

phys}cpu">



</gr:group>

<gr:group schema="os_network" name="AutoGroup_${OS-typ}_${ip-

adr:netmask=/24}">


network</gr:description>

</gr:group>

<gr:group schema="network_cpu" name="AutoGroup_${CPU-cnt-

phys}cpu_${ip-adr:netmask=/24}">

<gr:description>Group defined by number of

CPUs</gr:description>

</gr:group>

Parameter Reference


<gr:group schema="os" name="AutoGroup_${OS-typ}">

<gr:description>Group defined by operating

system</gr:description>

</gr:group>

<gr:group schema="network" name="AutoGroup_${ip-adr:netmask=/24}">

<gr:description>Group defined by network</gr:description>

</gr:group>

<gr:group schema="cpu" name="AutoGroup_${CPU-cnt-phys}cpu">

<gr:description>Group defined by number of

CPUs</gr:description>

</gr:group>



</gr:groups>

</defs:definitions>

Parameter Reference


7.4 Service Classes

The service classes are defined and parameterized in the group configuration file (my-

AMC_FA_Groups.xml) of a virtual FlexFrame pool.

A sevice class is defined by the following variables:

"system-id" ("P46", "O20", ...)

"service-type" ("db", "app", "ci", ...)

"service-id" ("00", ...)

The attributes service-priority and service-powervalue are defined in accord-

ance with these variables.

In the future it will be possible to extend such a service class by further attributes which,

for example, define the operating system required by a service or the number of

CPUs/performance requirement of the service.

7.4.1 Service Priority

The highest service priority is 1. Every service is assigned this priority by default, i.e. if no

service classes are defined, all services have the priority 1. The higher the number, the

lower the priority of a service.

Priority 0 has a special status. Setting priority 0 for a service class enables the autono-

mous functions to be disabled for a service.

The service priority is evaluated for all autonomous reactions. If, for example, a service of

a productive system and a service of a test system are running on the same node and the

test system’s service is assigned priority 5, this service is not executed because the pro-

ductive system’s service which is functioning without error has the higher priority of 1.

7.4.2 Service Power Value

The service power value specifies for a service a performance number which defines the

maximum performance (SAPS) required by this service. This is provided for future en-

hancements in the field of “load distribution and load transfer“.

A failed service with a performance value of 50 can, for example, also be taken over by a

node which still has at least 50 of its maximum performance number free.

7.4.3 Class Creation Rules

A service belongs either to the default class which always exists or it can be assigned

unambiguously to another class by evaluating the aforementioned variables.

7.4.4 Example

<gr:services>

Parameter Reference


<!—

"service schema" is specified in myAMC_FA.xml

allowed categories:

"system-id" ("P46", "O20", ...)

"service-type" ("db", "app", "ci", ...)

"service-id" ("00", ...)

...

-->



<value>DB</value>

<value>db</value>

</match>




</gr:service>



<value>CI</value>

<value>ci</value>

</match>




</gr:service>



<value>APP</value>

<value>app</value>

</match>




</gr:service>

... ...

7.5 FlexFrame Autonomy

The base parameterization is always required.

The parameterization is organized hierarchically, i.e. information can be configured iden-

tically for all services or specifically for individual services.

myAMC_FA.xml

The information to be configured relates to the following components:

Parameter Reference


General parameters

Parameters for the Performance and Accounting option

Node-related parameters

Service-related parameters

Path configurations

7.5.1 General Parameters

CheckCycleTime

Defines the cycles in which the detector modules supply results and the rule modules

evaluate the status derived from these. The parameter value may not be less than

the minimum processing time which the detector modules, rule modules and reaction

modules require to process a cycle. The default value in the as-supplied status is 10

seconds. The parameter value must also always be at least 1/3 of the lifetime of the

MonitorAlerts. In the FlexFrame standard installation the lifetime of the MonitorAlerts

is 30 seconds.

LivelistWriterTime

Defines the intervals at which the FA Agents must generate a Livelist. It is specified

in seconds.

ControlAgentTime

Specifies how often the Control Agent checks the Livelists of the Application Agents.

The parameter should thus be about the same as the LiveListWriterTime.

MaxHeartbeatTime

Specifies the maximum time which may elapse between two Livelist entries of an

Application Agent before the Control Agent intervenes. The MaxHeartbeatTime

must therefore always be greater than the ControlAgentTime and the Livelist-

WriterTime. In practice the factor of 3 between LivelistWriterTime and

MaxHeartbeatTime has proved practical.

MaxRebootTime

Specifies the maximum time which may elapse between two Livelist entries of an

Application Agent before the Control Agent intervenes if the latter is rebooting.

MaxFailedReachNumber

Specifies how often the Control Agent attempts to reach a node after the MaxHeart-

beatTime has been exceeded before an external SwitchOver is initiated.

MaxAgeSwitchOverFile

This parameter specifies the max age (in seconds) of SwitchOver-File.

If the age of a SwitchOver-file exceeds this value, the SwitchOver-file is ignored.

Node_SwitchOverTyp

Specifies the mode according to which the testaments are created: node-based or

service-based. The following keywords are currently valid for this parameter:

service

Parameter Reference


node

TakeOverStrategy

Defines how the application to take over services of defective nodes occurs and how

the winner is determined.

Allowed values are:

FirstFit

The first application node, which makes a entry to the switchover file, is

the winner and is allowed to take over the services.

LowPrioFit

The application node with the worst priority wins and is allowed to take

over the services. The priority of the application node will be taken from

the service with the highest priority on this application node.

Node_TakeOverRule

Defines which takeover rule is to be used. The following values are possible:

SpareNode

Substitution (add rule)

Displacement (replace rule)

Supplementation (supplementation)

Dynamic

Dyn_Spare_min

Dyn_Spare_max

Dyn_Add_min

Dyn_Add_max

Dyn_Replace_min

Dyn_Replace_max

Dyn_Substitute_mi

Dyn_Substitute_max

Dyn_Spare_exclusive

Defines the range of takeover rule “dynamic”.

CtrlAgent_WriteOnlyChanges

Controls the writing of output files.

The following values are possible:

false: files will be written every cycle (default) .

true: files will be written only if there was changes.

With this parameter load can be reduced on control node at great installations

and on the Storagesystem since changes are only written persistently at rele-

Parameter Reference


vant changes. The net load to the myAMC.FA WebGUI and the load the the

server part of the myAMC.FA WebGUI is also reduced.

7.5.2 Parameters for the Performance and Accounting Op-tion

FA Agent:

PerfdataReportCycleTime (myAMC_FA.xml)

The report cycle specifies the cycle of creating a performance and accounting value,

which is written to the collet.

max_colletcount_performance_files (myAMC_FA_AppAgent_spec.xml)

Number of collet generations that are stored before being rewritten.

max_filesize_performance_files (myAMC_FA_AppAgent_spec.xml)

Maximum size of the performance files. This parameter serves to limit the size of the

files.

collet_switch_start_performance_files (myAMC_FA_AppAgent_spec.xml)

Defines the date and time when new collet files are written.

Eexample: 2006-01-01 00:00:00

collet_switch_cycle_performance_files (myAMC_FA_AppAgent_spec.xml)

Cycle time when new collets are written. Example: 3600

By combining collet_switch_start_performance_files and

collet_switch_cycle_performance_files, it is possible to specify that, starting at

0:00, new collets are written every 3600 seconds, i.e. every hour.

It is also possible to implement different cycle times.

7.5.3 Parameters for the File System Utilization

Perfdata_FSUtil_ReportCycleTime

Specifies the report cycle time for FsUtil checks.

FSUtilScriptFilePath

Specifies the path to the FsUtil scripts.

Script file names are myAMC.DetGetDirData.sh, myAMC.DetGetFSData.sh.

Performance files Perf_FsUtil~<hostname>.prf.

Line-Identifier in this file is FSUTIL.

7.5.4 Configuration of the Spaceutil Functionality

The monitored directories and file systems will be pool specific configured in the configu-ration file myAMC_FA_Checks.xml in the directory

Parameter Reference


/opt/myAMC/vFF/vFF_<poolname>/config. The configuration consists of tem-

plates, depending on the actual conditions to be activated.

7.5.4.1 General structure

The configuration file for monitoring the filesystem utilization or the directory utilization

use a common structure. and a specific structure. For every Filesystem or directory to

monitor you have to define an entry in theConfigSection like “TrapTargets with the follow-

ing ConfigEntries :

Active

Indicates whether this entry is currently active or not. In case of inactive the complete

entry is ignored.

Name

Symbolic display name

The symbolic name can be freely assigned with a maximum lenth of 50 characters

and no special characters.

Group

Group name for logical grouping, that allows group based aggregation. The group

name can be freely assigned with a maximum length of 50 characters and no special

characters.

ContextObject

Reference object. Possible values are Frame, Pool, Group, Node, System, or

Service.

FrameActivationPattern

Wildcard pattern for frame-specifig activation

PoolActivationPattern

Wildcard patern for pool specific activation.

GroupActivationPattern

Wildcard Pattern for group specific activation.

NodeActivationPattern

Wildcard pattern for node specific activation

SystemActivationPattern

Wildcard pattern for system specific activation

ServiceTypeActivationPattern

Wildcard pattern for service specitifc activation

InstNumActivationPattern

Wildcard pattern for instance specific activation

The values in the reference object define the aggregation type of the data.

Parameter Reference


7.5.4.2 File System Utilization

In addition to the common values there are some attributes to define the parameters for

file system utilization.

Configuration set to monitor a file system (ConfigSection, like TrapTargets)

/myAMC.FA/FilesystemUtilization.

(ConfigEntries):

FilesystemNamePattern

Wildcard pattern to identify the file system.

If this value is missing or empty, this pattern is always true. This parameter can be definied with placeholder like $(name).

MountPointPattern

Wildcard pattern to select certain files using mount points.

If this value is missing or empty this value is always true. This parameter can be de-

finied with placeholder like $(name).

MaxUtilization

Thresholding (maximum utilization in percent) to trigger utilization events. The value

0 disables this threshold monitoring.

MaxGrowthRate

Threshold for maximum growth rate in percent per second. to trigger a trap as re-

sponse to large growth rates. The value 0 disables this threshold

7.5.4.3 Directory Utilization

Every directory to monitor need a config entry with following values (ConfigSection,

like TrapTargets) bellow the key /myAMC.FA/DirectoryUtilization.

In addition to the standard values (ConfigEntries) :

Paths

Absolute path of the directory being monitored.

This parameter can include placeholder in the form $(name).

7.5.4.4 Activation of Configration Entries

The directory and file system monitoring is based on a reference object. A reference object can be of the type Frame, Pool, Group, Node, System or Service.

The activation depends on the entry in the config files and dynamic values from the

agent.

Parameter name Activation depends on Example

Parameter Reference


Parameter name Activation depends on Example

AgentTypeActivationPattern Agent type

(“myAMC.AppAgent”,

“myAMC.FrameAgent”)

*Agent*

FrameActivationPattern Frame name

(allways local)

Local

PoolActivationPattern Pool name MyPool*

GroupActivationPattern Group name GR_Lin*

NodeActivationPattern Node name bx*

SystemActivationPattern SID P*

ServiceTypeActivationPattern Service type, possible

values: SRV_DBORA,

RV_DB2, SRV_DBSAP,

SRV_CI, SRV_APP,

SRV_J, SRV_JC,

SRV_SCS, SRV_ERS,

SRV_ASCS

SRV_DB*

InstNumActivationPattern Instance number

to definie a special in-

stance

02

For every config entry the FA AppAgent or FA FrameAgent checks together with the

dynamical information Framename, Poolname, Groupname, Nodename, Servicename if

there is a match with a config entry. So managed and monitored sets of objects are

dynamicly defined.

Reference objects Frame, Pool, Group, Node

Frame, pool, group or node specific rules to monitor file system utilization.

Reference object: Service.

The Service type ist the activator for a rule, for example monitor only spezial files if

their is an db service on the node.

Reference object System

The system name is activator for a rule

Examples:

AppAgent

Active Services: dbo46 (ora), dbp52 (sapdb), app02p52, cip55

Reference object: Group

Active: false

The entry is ignored because active is false.

Parameter Reference


AppAgent

Active Services: dbo46 (ora), dbp52 (sapdb), app02p52, cip55

Reference object: Pool

Active: true

ServiceTypeActivationPattern: SRV_DBORA

other ActivationPatterns: empty or „*“

The FA AppAgent activates a rule for the defined, but only for DB services.

AppAgent Active Services: dbo46 (ora), dbp52 (sapdb), app02p52, cip55

Reference objekt: System

Active: true

SystemActivationPattern: P*


The FA AppAgent activates a rule for SAP SIDs which match with P*.

FrameAgent Active Pools: Cust_1, pool2, Cust_3

Reference objekt: Pool

Active: true

FrameActivationPattern: A00*

PoolActivationPattern: Cust_*


The FA FrameAgent activate a monitoring rule for the pools Cust_1 and Cust_3,

but only if the name of the frames correlates with the FrameActivationPattern

7.5.4.5 Replaceing of Placeholders

The configuration section allows for some parameters the use of placeholders. The

placeholders a resolved from the FA AppAgent or FA FrameAgent. If a placeholder is not

resolved, an empty string is used.

The following placeholder are available:

${framename} (currently always Local)

Example: Local, A00001

${poolname}

Example: Cust_3, pool1

${groupname}

Example: GR_Linux

${nodename} Example: tombx2, sno1aplp2

${systemname}

Example: P46, ABC

Parameter Reference


${servicetype}

Example: SRV_APP, SRV_CI, SRV_DBORA, SRV_DBSAP, SRV_SCS, ...

${servicedisplaytype} Example: app, ci, db, scs, ...

${instnum}

The instance number will only be dissolved if it is valid (>=0). The number is always used

as double digit.

Example: 7 07

-1 “”

42 42

7.5.4.6 Wildcard Patterns

Wildcard patterns allow comparison wtih variable texts. Unknown elements can be se-

lected by the wildcard “*” and “?”.

“*” stands for an unlimited quantity of any character that may be present or not.

“?” stands for a single character

Example:

*.doc

matches with abc.doc, .doc and aabbccdd.doc , but not with abcdoc,

abc.doc2 or abc.xdoc

app??O46

matches with app1O46, app02O46, appO46 but not with app123O46, ciO46 or

ap02O46

app??P*

matches with app1P52, app02P5, appPPP but not with app123O46, app1O52, or

ap02O46

Parameter Reference


7.5.4.7 Configuration File myAMC_FA_Checks.xml




<configuration>

<configsection name="myAMC.FA">

<configsection name="Additional_Checks">





<configsection name="DirectoryUtilization">



<configsection name="FAPoolData">

<configentry name="Active">

<value

type="Boolean">false</value>

</configentry>



<configentry name="Name">

<value

type="String">fapooldata</value>

</configentry>

<configentry name="Group">

<value

type="String">fapooldata</value>

</configentry>

<configentry name="ContextObject">

<value type="String">Pool</value>

</configentry>



<configentry

name="AgentTypeActivationPattern">

<value

type="String">*Agent*</value>

</configentry>

<configentry

name="FrameActivationPattern">

<value type="String">*</value>

</configentry>

<configentry name="PoolActivationPattern">


</configentry>

<configentry

name="GroupActivationPattern">


</configentry>

<configentry name="NodeActivationPattern">


</configentry>

<configentry

name="SystemActivationPattern">


</configentry>

<configentry

name="ServiceTypeActivationPattern">


</configentry>

<configentry

name="InstNumActivationPattern">

Parameter Reference



</configentry>



<configentry name="Paths">

<value

type="String">/opt/myAMC/vFF/vFF_${poolname}/data</value>

</configentry>

</configsection>



<configsection name="OracleDB">


<value

type="Boolean">false</value>

</configentry>



<value

type="String">oradata</value>

</configentry>


<value

type="String">database</value>

</configentry>


<value

type="String">Service</value>

</configentry>



<configentry


<value

type="String">*AppAgent*</value>

</configentry>

<configentry



</configentry>



</configentry>

<configentry



</configentry>



</configentry>

<configentry



</configentry>

<configentry


<value

type="String">SRV_DBORA</value>

</configentry>

<configentry



</configentry>

Parameter Reference


<configentry name="Paths">

<value

type="String">/oracle/${systemname}/sapdata*</value>

</configentry>

</configsection>

</configsection>





<configsection name="FilesystemUtilization">



<configsection name="volFF">


<value type="Boolean">true</value>

</configentry>



<value type="String">volFF</value>

</configentry>


<value

type="String">storage</value>

</configentry>


<value type="String">Frame</value>

</configentry>



<configentry


<value

type="String">*FrameAgent*</value>

</configentry>

<configentry



</configentry>



</configentry>

<configentry



</configentry>



</configentry>

<configentry



</configentry>

<configentry



</configentry>

<configentry

Parameter Reference


</configentry>



<configentry name="FilesystemNamePattern">

<value

type="String">*:/*volFF*</value>

</configentry>

<configentry name="MountPointPattern">


</configentry>



<configentry name="MaxUtilization">

<value type="Float">90.0</value>

</configentry>

<configentry name="MaxGrowthRate">

<value type="Float">5.0</value>

</configentry>

</configsection>

</configsection>

</configsection>

</configsection>

</configuration>

Parameter Reference


7.5.5 Parameters for the Admin Pool Option

CtrlAgentRole

Allowed values are:

Normal Normal function of this CtrlAgent (don't work with a

caretaker of the admin pool).

WorkWithSparePool CtrlAgent work with a caretaker of the admin pool.

SparePoolName

Specifies the name of spare pool.

Allowed values are:

mySparePoolName Name of admin pool is mySparePoolName.

NONE No Admin Pool available.

If the own pool name is identical to the name of the admin pool, CtrlAgent of this pool

is the admin pool caretaker.

SpareGroupName

Specifies the name of the group in the admin pool.

Allowed values are:

mySpareGroupName Name of the group in admin pool is mySpareGroupName.

NONE No group in admin pool available.

SpareNodeRequestPath

Specifies the path for spare node requests.

MaxAgeResponseFile

Specifies the max age (in seconds) of response file (used by the AppAgent).

SpareRequestMoveTimeout

Timeout (in seconds) for moving an Application Node from the admin pool to work

pool (fa_move_node.sh -> ff_an_adm.pl).

SpareRequestMoveNodeMaxRetry

Max retries to call move-node script in case of error (LDAP-locks).

SpareRequestMoveNodeRetryDelay

Delay-Time (in seconds) between retries the call of move-node sript.

SpareNodePowercycleTime

Cycle time (in minutes) how often the spare node in spare-pool will be switches ON

and OFF.

Values <= 1 are forbidden and switch off this functionality.

SelectionRuleConfigFile

Path name and file name of the config file for he selection of the spare nodes from

the admin pool.

Parameter Reference


7.5.6 Parameters for ServicePing

ServicePingScriptFilePath

This parameter specifies the path of the ServicePing scripts. The script names are

ServicePingDb.sh, ServicePingDef.sh and ServicePingErs.sh.

ServicePing_is_active

This parameter switches on and off ServicePing functionality respectively.

To switch on ServicePing functionality, this parameter must be set to true and also

the return code of the ServicePing scripts must be changed. The return code is 99 by

default and means unknown.

CheckCycle_of_ServicePing

This parameter defines the check cycle (in seconds) of the ServicePing functionality.

7.5.7 Node-related Parameters

Node_MaxRebootNumber

Specifies how many consecutive reboots may be performed to restore a service. If 3

is specified, the Application Agent attempts to make the system available again with

up to three reboots. Here you must bear in mind that a reboot is also unsuccsessful if the system could not be restored within the MaxRebootTime set. In the event of re-

boot problems, the MaxRebootTime parameter must therefore also always be

checked and compared with the reboot time actually needed.

Node_MaxSwitchOverNumber

Specifies how many consecutive switchovers may be performed to restore a service.

Node_SwitchOverServiceStartDelayTime

After an internal SwitchOver this parameter defines how long will be waited until the

services to be taken over are started on the node that is taking over control. This val-

ue is needed since, depending on the switches used, the virtual IP addresses cannot

be released directly owing to the devices’ internal caching. This delay time must

therefore be greater than the devices’ caching time otherwise the service startup on

the node which is taking over control will fail.

Node_SendTrapsAllowed

Releases or blocks the sending of node traps.

Node_RebootCommand

Specifies which command is executed when the Application Agent initiates a reboot.

Normally this is a shutdown with a subsequent reboot.

Node_ShutdownCommand

Specifies which command is executed when the Application Agent initiates a switch-

over. Normally this is a shutdown without a subsequent reboot.

Parameter Reference


Node_PowerDownCommand

Specifies which command is executed by the Control Agent before an external

SwitchOver is initiated. In this way it is ensured that the services on the node being

switched over are really stopped and can be taken over without any problem by other

nodes. The Control Agent waits at most for the period specified with the

ShutdownFac_Shut_Exec_Timeout parameter before it continues with the

switchover.

Node_CheckAvailabilityCommand

Specifies which command is executed by the Control Agent to check the availability

of a node. A return value of 0 is interpreted as a positive result, every other return

value as negative. The Control Agent waits at most for the period of 5 seconds. If the

command has not been executed completely by then, it is assumed that the test is

negative, i.e. the node is no longer available, resulting in an external SwitchOver.

Node_RemoteExecutionCommand

Specifies which command the Control Agent puts ahead of a command to be execut-

ed on another node. This is used, for example, to start or stop a service remotely on

an Application Node. Usually ssh is used here.

SwitchOver_ext_Unavailability_check

Specifies which unavailability check will be performed before external SwitchOver.

Possible values are:

PING_and_SSH ping and ssh must fail for Poff (default)

PING_or_SSH ping or ssh must fail for Poff

PING_only ping only must fail for Poff

SSH_only ssh only must fail for Poff

SwitchOver_ext_Unavailability_check_PING

Specifies which ping unavailability check will be performed before external Switch-

Over. Possible values are: 1 ping the normal host name (blade2) (default)

2 ping the server LAN host name (blade2-se)

4 ping the storage LAN host name (blade2-st)

3 ping normal and server LAN

5 ping normal and storage LAN

6 ping server-lan and storage LAN

7 ping normal and server LAN and storage LAN

7.5.8 Service-related Parameters

The following parameters can be set individually for each service type or for multiple

services simultaneously. This is also a result of the hierarchical structure of the parameter

file. In the parameter file there is also an option for configuring the values for the DB, CI

and APP services individually. The value of the default service is used for any value

which is not service-specific.

Parameter Reference


Service_EnableMonitoring

Service_SendTraps


Service_TrapSendDelayTime

Service_ReactionDelayTime

Service_MaxStartTime

Service_MaxStopTime

Service_PingVirtualServiceInterface

Service_ReactionScript

The dynamic behavior of the AppAgents and CtrlAgents depends very much on the val-

ues in the configuration file and the physical conditions. You must therefore check very

carefully that the relation between certain values is secure and application-oriented.


Defines whether monitoring is enabled or disabled for the service type in question.

Service_SendTraps

Releases or blocks the sending of service traps.


Defines how many attempts are made to restart a failed service. This value can be

configured individually for each service type. The value is typically in the range 1 to

10. The value 0 means that no attempt is made to restart a failed service. If reboots

are permitted on the node, failure of a service leads directly to a reboot.


Defines the send delay time for the service traps.


Interworks directly with CheckCycleTime. It can be set individually for each service

type. This time defines how long the triggering of a reaction is delayed after a failure

has been detected.

Examples:

CheckCycleTime = 10 sec; ServiceReactionDelayTime = 30 sec

In this example a failed service is detected in a cycle. However, the reaction only

takes place after 30 seconds. The failure must therefore have been identified as a

failure over at least three detection cycles. This allows you to prevent a detection er-

ror resulting in an incorrect reaction.

Parameter Reference


CheckCycleTime = 10 sec; ServiceReactionDelayTime = 0 sec

In this example the required reaction takes place immediately in the cycle in which

the problem was detected.

Service_MaxRestartTime

Defines the maximum time which may be required for a service type in the event of a

restart. If this time is exceeded, a second or nth attempt is made in accordance with Service_MaxRestartNumber. Thus if too short a time is selected for the service

to be monitored and the hardware used, i.e. the service requires longer to restart

than permitted by Service_MaxRestartTime, a problem situation is triggered in-

correctly.


Defines how long a service may take to start up. If this time is exceeded, the Agent

interprets the service as not started and initiates further reactions.

Service_MaxStopTime

Defines how long a service may take to stop. If this time is exceeded, the Agent in-

terprets the service as not stopped and initiates appropriate reactions.

Service_PingVirtualServiceInterface

Defines whether the associated virtual FlexFrame service interface is pinged to de-

termine the availability of a service. If it is set to 0 the virtual LAN interfaces of the cli-

ent and server network are not queried. Interface availability then has no influence on

the status change of a service. Switching off this parameter causes a defined wait

time (which is parameter Node_SwitchOverServiceStartDelayTime) before

starting services in a takeover (switchover).

Service_ReactionScript

Defines the interace to the FF-SAP-Start-/Stop-Scripts.



<value

type="String">${ScriptPath}/${ScriptPrefix}${monparam}${ScriptPostfix}

${BYNIDENT} ${SIDENT} ${ACTION}</value>



<value type="String">${ScriptPath}/ff_service.sh -s ${SIDENT} -t

${monparam} ${BYKEYNIDENT} -a ${ACTION}</value>

7.5.9 Parameters for the Definition of a Generic Service

By default, the FA Agents can supervise a defined quantity of services, depending on the

version. For these services, the rules for the detection and the rules for the autonomous

reaction are fixed components of the FA Agents. Generic services are services which are

not considered in the standard scope of the FA Agents but which will also be considered

within the context of supervision and of the autonomous reaction.

Parameter Reference


This section describes the parameters that are used to enable monitoring by the FA

Agents as well as the autonomous reactions.

In the parameter file myAMC_FA_Rules.xml, the necessary information for the detection

of the service is registered. Most of the parameters are optional, meaning they are not

normally needed for monitoring. In the following the mandatory parameters are indicated

by an asteriks (*). The optional parameters are set with default values if not used.

The parameter file for the definition of generic services is organized in two main parts: the

parametering of the detection and of the reactions which are required for a service.

For the configuration of a generic service, this generic service must be described in the

detection section.The reaction is then defined in the second section.

Therefore, ideally, in each case one of the templates in the detection and reaction block

is copied. This must then be individualized with the specific parameters in the appropiate

places. The referencing between the detection and reaction blocks occurs via the defined

service name.

Attention: The templates are set to ‘not active’. Therefore, in principle, the individualiza-tion of the parameter Active must be changed to 1.

Furthermore, the standard service parameters (see section above) must be defined such

that they are also valid for built-in services. If service-specified parameters are not set,

the default values will be used.


Service_SendTraps





Service_MaxStopTime

7.5.9.1 Parametering of the Service Detection

The parameter set for the detection consists of a header which defines the service as a

whole. A service itself can consist of one or more subservices. For every subservice a

separate subservice parameter block must be created.

7.5.9.1.1 Parameters of the service header

*Name

Unique symbolic servicename inside the rulefile. The maximum string length is 50

Parameter Reference


characters. It does not have to be identical with the technical service name. The ser-

vice name is the reference to the reaction.

Description

Used for documentation purposes only.

*Displayname

The display name is used by the FlexFrame ControlCenter. If it is not defined, the

symbolic name is used, but it should be defined. It should be as short as possible be-

cause this service name will be displayed in the FlexFrame ControlCenter (limited

space available).

Active

Activates (1) or deactivates (0) the parameter block.

MonitorParam

Defines the required parameters that produce a state-altering event during starting or

stopping of a service. The event script is called MonitorFlag and is doing nothing.

It remains in the pro-cess list for a defined timeperiod (normally 30 seconds).To at-tach the MonitorFlag to a service, the following call syntax is prescribed:

MonitorFlag <RefSrv> <State> <InstNo> <SID>

RefSrv

This is a reference to a service and is defined under MonitorParam.

State

start, stop, restart, watch or nowatch

InstNo

A positive instance number; this information is optional

SID

The system id. This information is optional

Orderprio

When there are several services, the order priority defines within a system the order

in which they will be started. Stopping will be done in the reverse order.

PowerValue

Workload demands of a service in SAPS.

ServicePrio

Priority of the service to be defined.

*Group

Accounting group name for the performance and accounting management. A new

group can be defined for a service or its values appear in an existing group.

Parameter Reference


For example, SAP: If the backup server should be monitored and its workload con-

sumption should appear in the group SAP, then it should be configured to the group

SAP.

7.5.9.1.2 Subservice parametering

A service consists of a minimum of one or more subservices. For every subservice the

following parameters must be defined:

Subservice

Subservice name

Symbolic name for the subservice. This parameter is optional.

Display name

This parameter is for example used as service name in failure traps or error messag-

es.

Active

Active (1) or inactive (0). The default is active.

7.5.9.1.3 Subservice detector parametering

For every subservice one or more detectors can be activated. In version 3.0 of the FA

Agents, only detectors of the type process exist. In future versions more detectors will be

added. A detector requires the following parameters:

Detector

Detector type

Process

Active

Switches the rule on or off

ProcessName

The process name to be detected

CountMin

Minimum number of required processes of this type

CountMax

Maximum number of required processes of this type

HierachyMin

Minimum process hierachy of the subservices

Parameter Reference


Severity

Determined Severity (warning or critical) if the affiliated subservice has a fault.

7.5.9.2 Parametering of the Service Reactions

The second big block to be defined is the reactions required for a service. For this the

commands for starting, stopping and restarting the services must be defined.

The reaction parameter block for a service also has the service name, meaning the same

name. In a reaction the determining of a program or script to be called occurs in the case

of each start, stop or restart of the services.

The first command specifies the start call, the second the stop call and the third the re-

start call.

Each command is composed of the attributes script and parameter. script defines

the called program or script. If a parameter is omitted, no parameter is used.

Parameter Reference


7.5.10 Path Configuration

The path configuration is used to define the directories in which the myAMC.FA compo-

nents store their various work files. A FlexFrame Autonomy solution stores a range of

information, such as files with display information for the WebInterface and logging infor-

mation to be used for support when this is required, in various files.

To ensure performance and clarity are retained even in larger configurations, we

recommend that you do not modify these settings! If the suggested path configura-

tion is changed, though, make sure that clatiy is still retained and no problems

arise with regard to performance and accessibility.

LiveListLogFilePath

This parameter specifies the directory in which the Livelist is stored.

LiveListXmlFilePath

This parameter specifies the directory in which the XML representation of the Livelist

is stored. This file is required by the FA WebInterface. The parameter should contain

the same path as ServicesXmlFilePath.

ServicesXmlFilePath

This parameter specifies the directory in which the XML representation of the ser-

vices list is stored. These files are required by the FA WebInterface. The parameter

should contain the same path as LiveListXmlFilePath.

ServicesListFilePath

This parameter specifies the directory in which the services list files are stored.

ServicesLogFilePath

This parameter specifies the directory in which the services log files are stored.

RebootListFilePath

This parameter specifies the directory in which the reboot files are stored. These files

contain a list of all services which must be restored after a reboot.

SwitchOverListFilePath

This parameter specifies the directory in which the switchover files are stored. These

files contain a list of all services which must be restored on another node after a

switchover.

PerformanceFilePath

This parameter specifies the directory in which the performance files are stored.

These files contain measured values for performance data.

FSUtilScriptFilePath

Specifies the path to the FsUtil-scripts.

Script-file names are 'myAMC.DetGetDirData.sh', 'myAMC.DetGetFSData.sh'.

Performance files 'Perf_FsUtil~<hostname>.prf'.

Line-Identifier in this file is 'FSUTIL'.

Parameter Reference


SAPScriptFilePath

This parameter specifies the directory in which the start and stop scripts for the SAP

services (sapdb, sapci, sapapp, etc.) can be found. The default path

(/opt/myAMC/scripts/sap) is normally a symbolic link to the actual script directo-

ry.

ControlFilePath

This parameter specifies the directory in which the control files (<service

type><service id><service sid>_host) generated by the start/stop scripts

are contained.

BlackboardFilePath

This parameter specifies the directory in which the BlackBoard file can be found.

Commands can be entered in it which are executed by the FA_AppAgents.

GroupConfigFile

This parameter specifies the file in which the group affiliation is configured.

PrePoffHookPath

This parameter specifies the script, which is execute before powering off a node. If

IgnorePoffHookResult = true, the return code will be ignored, otherwise the

node will be powered off only if this script returns 0.

PostPoffHookPath

This parameter specifies the script, which is execute after powering off a node and

before performing a SwitchOver. If IgnorePoffHookResult = true, the return

code will be ignored, otherwise the SwitchOver will be performed only if this script re-

turns 0.

ServicePingScriptFilePath

This parameter specifies the path of the serviceping scripts. The names of the called

scripts are 'ServicePingDb.sh', 'ServicePingDef.sh' and 'ServicePingErs.sh'.

Parameter Reference


7.5.11 Shutdown Configuration

The shutdown feature is described in detail in section 6.4.

IgnoreShutdownFailure

The parameter IgnoreShutdownFailure defines whether, after a failed shut-

down/powerdown, the network interfaces of the relevant node are to be deactivated,

which would ensure that all services running on this node are shut down and the ser-

vices can be switched over to an other Application Node.

Shut_DiagTrapRepeat_valid

Time in seconds to resend the „shutdown validity is valid“ – trap.

(‚0‘ == no resend)

Shut_DiagTrapRepeat_invalid

Time in seconds to resend the „shutdown validity is invalid“ – trap.

(‚0‘ == no resend)

Parameter Reference



Parameter Reference





<configuration>

<configsection name="myAMC.FA">





<configentry name="CheckCycleTime">

<value type="UnsignedInteger">10</value>

</configentry>



<configentry name="BlackBoardCheckCycleTime">


</configentry>



<configentry name="PersistFAInfoCheckCycleTime">


</configentry>



<configentry name="LivelistWriterTime">


</configentry>



<configentry name="LivelistRead_dT">


</configentry>



<configentry name="CheckCycle_of_ServicePing">


</configentry>



<configentry name="ServicePing_is_active">

<value type="Boolean">false</value>

</configentry>



<configentry name="CtrlAgent_WriteOnlyChanges">


</configentry>

Parameter Reference




<configentry name="VirtualColdStandbySpareNode_use">


</configentry>



<configentry name="CtrlAgentRole">



<value type="String">Normal</value>

</configentry>



<configentry name="SparePoolName">

<value type="String">NONE</value>

</configentry>



<configentry name="SpareGroupName">

<value type="String">SpareGrp</value>

</configentry>





<configentry name="SpareRequestMoveTimeout">


</configentry>



<configentry name="SpareRequestMoveNodeMaxRetry">


</configentry>



<configentry name="SpareRequestMoveNodeRetryDelay">


</configentry>



<configentry name="SpareNodePowercycleTime">


</configentry>



<configentry name="SRDF_NAS_FH_NASCheck_Timeout">

Parameter Reference



</configentry>



<configentry name="SRDF_NAS_FH_MaxSRDF_RepairTime">


</configentry>



<configentry name="SRDF_NAS_FH_CheckTrigger">



<value type="String">NO_Check</value>

</configentry>



<configentry name="SRDF_NAS_FH_Reaction">



<value type="String">manual</value>

</configentry>

<!-- specifies how often (in seconds) myAMC.FA looks for new live list

Parameter Reference


entries.

This value should not exceed a third of the time interval specified by the

parameter MaxHeartbeatTime.

-->

<configentry name="ControlAgentTime">


</configentry>



<configentry name="MaxHeartbeatTime">


</configentry>





<configentry name="MultiNodeFailure_ShortTime_FailureCount">


</configentry>



<configentry name="MultiNodeFailure_ShortTime_FailureTime">


</configentry>



<configentry name="MultiNodeFailure_ShortTime_ReactionDelay">


</configentry>



<configentry name="MultiNodeFailure_ShortTime_ReactionAction">



<value type="String">None</value>

</configentry>





<configentry name="MultiNodeFailure_LongTime_FailureCount">


</configentry>



<configentry name="MultiNodeFailure_LongTime_FailureTime">


</configentry>


in case of 'MultiNodeFailure_LongTime'. -->

<configentry name="MultiNodeFailure_LongTime_ReactionDelay">


</configentry>



<configentry name="MultiNodeFailure_LongTime_ReactionAction">




</configentry>





<configentry name="MultiNodeFailure_Heartbeat_FailureCount">


</configentry>



<configentry name="MultiNodeFailure_Heartbeat_FailureTime">

Parameter Reference


</configentry>


in case of 'MultiNodeFailure_Heartbeat'. -->

<configentry name="MultiNodeFailure_Heartbeat_ReactionDelay">


</configentry>



<configentry name="MultiNodeFailure_Heartbeat_ReactionAction">




</configentry>



<configentry name="MaxRebootTime">


</configentry>



<configentry name="MaxUpTime_for_Reboot_detection">


</configentry>



<configentry name="MaxFailedReachNumber">


</configentry>



<configentry name="SwitchOver_ext_Unavailability_check">



<value type="String">PING_and_SSH</value>

</configentry>



<configentry name="SwitchOver_ext_Unavailability_check_PING">


</configentry>

Parameter Reference


<configentry name="PerfdataReportCycleTime">


</configentry>



<configentry name="Perfdata_SpaceUtil_FSCycleTime">


</configentry>



<configentry name="Perfdata_SpaceUtil_DirCycleTime">


</configentry>



<configentry name="MaxAgeSwitchOverFile">


</configentry>



<configentry name="MaxAgeResponseFile">


</configentry>





<configentry name="ComMethod_LiveList">

<value type="String">File-based</value>



</configentry>



<configsection name="Node">

Parameter Reference


<configentry name="Node_ServiceSchema">

<value type="String">default</value>

</configentry>





<configentry name="Node_GroupSchema">

<value type="String">default_LDAP</value>

</configentry>



<configentry name="Node_SAP_Interface">

<value type="String">SAPscripts</value>





</configentry>



<configentry name="Node_SAP_Interface_ReplaceSTR_ScriptPath">

<value type="String">${ScriptPath}</value>

</configentry>

<configentry name="Node_SAP_Interface_ReplaceSTR_ScriptPrefix">

<value type="String">${ScriptPrefix}</value>

</configentry>

<configentry name="Node_SAP_Interface_ReplaceSTR_ScriptPostfix">

<value type="String">${ScriptPostfix}</value>

</configentry>



<configentry name="Node_SAP_Interface_ScriptPrefix_SAP">

<value type="String">sap</value>

</configentry>

<configentry name="Node_SAP_Interface_ScriptPrefix_ACC">

<value type="String">acc</value>

</configentry>



<configentry name="Node_SAP_Interface_ScriptPostfix_SAP">


</configentry>

<configentry name="Node_SAP_Interface_ScriptPostfix_ACC">


</configentry>





<configentry name="Node_SwitchOverTyp">



<value type="String">node</value>

</configentry>




= 'SpareNode':

Only a spare node will Start the services from taken

SwitchOver-File.

= 'add' (Hinzufuegen, Ergaenzung):

Additional start the services from taken SwitchOver-File.

= 'replace' (Verschieben):

Running services will be stopped and a SwitchOver-File will

be written therefore,

Start the services from taken SwitchOver-File.

= 'substitute' (Ersetzung):

Parameter Reference


Running services will be stopped,

Start the services from taken SwitchOver-File.

= 'dynamic':

Cause of prios there will be done one of the TakeOver-rules

'SpareNode', 'add', 'replace', 'substitute', or none.

-->

<configentry name="Node_TakeOverRule">



<value type="String">SpareNode</value>

</configentry>





<configentry name="Dyn_Spare_min"> <value

type="UnsignedInteger">1</value> </configentry>

<configentry name="Dyn_Spare_max"> <value




<configentry name="Dyn_Add_min"> <value


<configentry name="Dyn_Add_max"> <value




<configentry name="Dyn_Replace_min"> <value


<configentry name="Dyn_Replace_max"> <value




<configentry name="Dyn_Substitute_min"> <value


<configentry name="Dyn_Substitute_max"> <value




<configentry name="Dyn_Spare_exclusive">


</configentry>




= 'Node':

Node-escalation: "Restart" => "Reboot" =>

"SwitchOver"

The escalation will be done for all services on a

node.

= 'Service':

Service-escalation: "Restart" =>

"SwitchOver"(single service)

The escalation will be done for a single service.

-->

<configentry name="EscalationType">



<value type="String">Node</value>

</configentry>

Parameter Reference





= 'FirstFit':

The first node, who applies for TakeOver, wins

and gets the SwitchOver-File.

= 'LowPrioFit':

The node with the minimal prio, wins and gets the

SwitchOver-File.

The lowest Service-priority you may use is '9999'

(As default a SpareNode has a prio of '10000').

-->

<configentry name="TakeOverStrategy">

<value type="String">FirstFit</value>



</configentry>



<configentry name="Node_MaxRebootNumber">


</configentry>



<configentry name="Node_MaxSwitchOverNumber">


</configentry>



<configentry name="Node_SwitchOverServiceStartDelayTime">


</configentry>



<configentry name="Node_SendTrapsAllowed">


</configentry>



<configentry name="Node_RebootCommand">

<value

type="String">/opt/myAMC/scripts/ShutDown_Node/shutdown_node.sh reboot</value>

</configentry>



<configentry name="Node_ShutdownCommand">

<value

type="String">/opt/myAMC/scripts/ShutDown_Node/shutdown_node.sh shutdown</value>

</configentry>



<configentry name="Node_PowerDownCommand">



<value type="String">/bin/su - root -c

"/opt/myAMC/scripts/PowerMng/powerOnOff.sh ${node-name} down"</value>

Parameter Reference


</configentry>



<configentry name="Node_CheckAvailabilityCommand">

<value type="String">/usr/bin/ssh -q ${node-name}

/bin/uname -a</value>

</configentry>



<configentry name="Node_RemoteExecutionCommand">

<value type="String">/usr/bin/ssh -q root@${node-name}

</value>

</configentry>

</configsection>



<configsection name="Services">

<configsection name="Default">



<configentry name="Service_EnableMonitoring">


</configentry>



<configentry name="Service_SendTraps">


</configentry>



<configentry name="Service_MaxRestartNumber">


</configentry>



<configentry name="Service_TrapSendDelayTime">


</configentry>



<configentry name="Service_ReactionDelayTime">


</configentry>



<configentry name="Service_MaxStartTime">


</configentry>



<configentry name="Service_MaxStopTime">


</configentry>



<configentry name="Service_PingVirtualServiceInterface">


</configentry>

Parameter Reference







<value type="String">${ScriptPath}/ff_service.sh

-s ${SIDENT} -t ${monparam} ${BYKEYNIDENT} -a ${ACTION}</value>

</configentry>

</configsection>

<configsection name="SRV_ORACLE_RAC_CRS">




</configentry>



</configentry>


>



</configentry>




</configentry>




</configentry>

</configsection>

<configsection name="SRV_ORACLE_RAC_DB">




</configentry>



</configentry>


>



</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_DBORA">





</configentry>





</configentry>

Parameter Reference




</configentry>

</configsection>


<configsection name="SRV_DBSAP">




</configentry>




</configentry>




</configentry>

</configsection>


<configsection name="SRV_DB2">




</configentry>




</configentry>




</configentry>







<value type="String">${ScriptPath}/ff_service.sh

-s ${SIDENT} -t db ${BYKEYNIDENT} -a ${ACTION}</value>

</configentry>

</configsection>



<configsection name="SRV_SMD73">




</configentry>




</configentry>




</configentry>





<value

type="String">${ScriptPath}/ff_smd_agent.sh -s ${SIDENT} -t db ${BYKEYNIDENT} -a ${ACTION}</value>

</configentry>

</configsection>

Parameter Reference




<configsection name="SRV_CI">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_APP">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_SCS">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_ASCS">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_JC">




</configentry>

Parameter Reference


</configentry>




</configentry>

</configsection>



<configsection name="SRV_J">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_LC">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_ERS">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_MDS71">




</configentry>




</configentry>




</configentry>

</configsection>

Parameter Reference


<configsection name="SRV_MDIS71">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_MDSS71">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_XPR">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_CMS">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_TRX">




</configentry>

Parameter Reference



</configentry>




</configentry>

</configsection>



<configsection name="SRV_BOE4">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_BOE_SRV">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_SMD">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_PAI">




</configentry>




</configentry>

Parameter Reference


</configentry>

</configsection>



<configsection name="SRV_VMWARE">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_CTM">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_WEBDISP">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_SAPSYBASE">




</configentry>




</configentry>




</configentry>

</configsection>



<configsection name="SRV_HANA">

Parameter Reference


</configentry>




</configentry>




</configentry>

</configsection>

</configsection>





<configentry name="FAScriptFilePath">

<value type="String">/opt/myAMC/scripts</value>

</configentry>



<configentry name="FAScriptFilePathLocal">

<value type="String">/opt/local/myAMC/scripts</value>

</configentry>



<configentry name="SpareNodeRequestPath">

<value type="String">/opt/myAMC/vFF/SpareRequest</value>

</configentry>



<configentry name="VirtualColdStandbySpareNodeFilePath">

<value type="String">/opt/myAMC/vFF/vFF_${vFF}/config</value>

</configentry>



<configentry name="LiveListLogFilePath">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/livelist</value>

</configentry>



<configentry name="LiveListXmlFilePath">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/xmlrepository</value>

</configentry>



<configentry name="ServicesXmlFilePath">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/xmlrepository</value>

</configentry>



<configentry name="ServicesListFilePath">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/servicelists</value>

</configentry>



<configentry name="ServicesLogFilePath">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/servicelogs</value>

Parameter Reference


</configentry>



<configentry name="RebootListFilePath">

<value type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/reboot</value>

</configentry>



<configentry name="SwitchOverListFilePath">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/switchover</value>

</configentry>



<configentry name="PerformanceFilePath">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/performance</value>

</configentry>



<configentry name="BlackboardFilePath">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/blackboard</value>

</configentry>



<configentry name="PersistFAInfoFilePath">

<value type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/reboot</value>

</configentry>



<configentry name="ShutdownCfgFilePath">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/shutdown</value>

</configentry>



<configentry name="ACCScriptFilePath">

<value type="String">/opt/myAMC/scripts/acc</value>

</configentry>



<configentry name="SAPScriptFilePath">

<value type="String">/opt/myAMC/scripts/sap</value>

</configentry>



<configentry name="ControlFilePath">

<value type="String">/opt/myAMC/scripts/sap/log</value>

</configentry>



<configentry name="ServicePingScriptFilePath">


</configentry>



<configentry name="FSUtilScriptFilePath">


</configentry>

<!-- specifies path to DetTest - scripts, output and timeout [s].

Parameter Reference


-->

<configentry name="DetTest_Sapparar_ScriptFilePath">



<value type="String">../myAMC.DetTestSapparar.sh</value>

</configentry>

<configentry name="DetTest_Sapparar_ScriptTimeout">


</configentry>

<configentry name="DetTest_Sapparar_OutputFilePath">

<value type="String">/tmp/myAMC.DetTestOutSapparar.txt</value>

</configentry>

<configentry name="DetTest_Ensmon_ScriptFilePath">



<value type="String">../myAMC.DetTestEnsmon.sh</value>

</configentry>

<configentry name="DetTest_Ensmon_ScriptTimeout">


</configentry>

<configentry name="DetTest_Ensmon_OutputFilePath">

<value type="String">/tmp/myAMC.DetTestOutEnsmon.txt</value>

</configentry>

<configentry name="DetTest_Dbmcli_ScriptFilePath">



<value type="String">../myAMC.DetTestDbmcli.sh</value>

</configentry>

<configentry name="DetTest_Dbmcli_ScriptTimeout">


</configentry>

<configentry name="DetTest_Dbmcli_OutputFilePath">

<value type="String">/tmp/myAMC.DetTestOutDbmcli.txt</value>

</configentry>





<configentry name="HostInfoLDAPFile">

<value type="String">/opt/myAMC/vFF/vFF_${vFF}/config/myAMC_FA_Hosts-

HOSTNAME.xml</value>

</configentry>



<configentry name="GroupConfigFile">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/config/myAMC_FA_Groups.xml</value>

</configentry>



<configentry name="RuleConfigFile">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/config/myAMC_FA_Rules.xml</value>

</configentry>



<configentry name="RuleCustomConfigFile">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/config/myAMC_FA_Rules_Custom.xml</value>

</configentry>

Parameter Reference




<configentry name="SelectionRuleConfigFile">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/config/myAMC_FA_SelectionRules.xml</value>

</configentry>





<configentry name="LanPostfixServer">

<value type="String">-se</value>

</configentry>



<configentry name="LanPostfixClient">


</configentry>



<configentry name="LanPostfixStorage">

<value type="String">-st</value>

</configentry>



<configentry name="LanPostfixControl">

<value type="String">-co</value>

</configentry>



<configentry name="RespectServiceDependencies">


</configentry>

<configsection name="Shutdown_Facility">

<configsection name="Executable">

<configentry name="Shut_Ex_Blade">


</configentry>

<configentry name="Shut_Ex_IPMI">


</configentry>

<configentry name="Shut_Ex_IPMIPOWER">


</configentry>

<configentry name="Shut_Ex_RSB">


</configentry>

<configentry name="Shut_Ex_XSCF">


</configentry>

<configentry name="Shut_Ex_RPS">


</configentry>

<configentry name="Shut_Ex_SCON">


</configentry>

<configentry name="Shut_Ex_XSCF2">


</configentry>

<configentry name="Shut_Ex_ALOM">


</configentry>

<configentry name="Shut_Ex_FA_BLADE">

<value

type="String">/opt/myAMC/scripts/PowerMng/blade_wrapper.sh</value>

Parameter Reference


</configentry>

<configentry name="Shut_Ex_FA_IPMI">

<value

type="String">/opt/myAMC/scripts/PowerMng/ipmi_wrapper.sh</value>

</configentry>

<configentry name="Shut_Ex_FA_XSCF2">


</configentry>

<configentry name="Shut_Ex_FA_XSCF">


</configentry>

<configentry name="Shut_Ex_FA_RPS">


</configentry>

<configentry name="Shut_Ex_FA_VM">

<value

type="String">/opt/myAMC/scripts/PowerMng/vm_vm_wrapper.sh</value>

</configentry>

<configentry name="Shut_Ex_FA_HV">

<value

type="String">/opt/myAMC/scripts/PowerMng/vm_hv_wrapper.sh</value>

</configentry>

<configentry name="Shut_Ex_FA_GENERIC">

<value

type="String">/opt/myAMC/scripts/PowerMng/generic_wrapper.sh</value>

</configentry>

<configentry name="Shut_Ex_RESERVED_1">


</configentry>



</configentry>



</configentry>



</configentry>



</configentry>

<configentry name="Shut_Ex_WINSTD">

<value type="String">Shutdown.exe</value>

</configentry>

<configentry name="Shut_Ex_WIN_RESERVED_1">


</configentry>



</configentry>



</configentry>



</configentry>



</configentry>

</configsection>

<configsection name="Configuration">

<configentry name="Shut_Cnf_Blade">


</configentry>

<configentry name="Shut_Cnf_IPMI">


</configentry>

<configentry name="Shut_Cnf_IPMIPOWER">


</configentry>

<configentry name="Shut_Cnf_RSB">


Parameter Reference


</configentry>

<configentry name="Shut_Cnf_XSCF">


</configentry>

<configentry name="Shut_Cnf_RPS">


</configentry>

<configentry name="Shut_Cnf_SCON">


</configentry>

<configentry name="Shut_Cnf_XSCF2">


</configentry>

<configentry name="Shut_Cnf_ALOM">


</configentry>

<configentry name="Shut_Cnf_FA_BLADE">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/shutdown/FA_blade.cfg</value>

</configentry>

<configentry name="Shut_Cnf_FA_IPMI">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/shutdown/FA_ipmi.cfg</value>

</configentry>

<configentry name="Shut_Cnf_FA_XSCF2">


</configentry>

<configentry name="Shut_Cnf_FA_XSCF">


</configentry>

<configentry name="Shut_Cnf_FA_RPS">


</configentry>

<configentry name="Shut_Cnf_FA_VM">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/shutdown/FA_VM.cfg</value>

</configentry>

<configentry name="Shut_Cnf_FA_HV">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/shutdown/FA_HV.cfg</value>

</configentry>

<configentry name="Shut_Cnf_FA_GENERIC">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/data/FA/shutdown/FA_Generic.cfg</value>

</configentry>

<configentry name="Shut_Cnf_RESERVED_1">


</configentry>



</configentry>



</configentry>



</configentry>



</configentry>

<configentry name="Shut_Cnf_WINSTD">


</configentry>

<configentry name="Shut_Cnf_WIN_RESERVED_1">


</configentry>



</configentry>



</configentry>


Parameter Reference



</configentry>



</configentry>

<configentry name="Shut_Prio_Blade">


</configentry>

<configentry name="Shut_Prio_IPMI">


</configentry>

<configentry name="Shut_Prio_IPMIPOWER">


</configentry>

<configentry name="Shut_Prio_RSB">


</configentry>

<configentry name="Shut_Prio_XSCF">


</configentry>

<configentry name="Shut_Prio_RPS">


</configentry>

<configentry name="Shut_Prio_SCON">


</configentry>

<configentry name="Shut_Prio_XSCF2">


</configentry>

<configentry name="Shut_Prio_ALOM">


</configentry>

<configentry name="Shut_Prio_FA_BLADE">


</configentry>

<configentry name="Shut_Prio_FA_IPMI">


</configentry>

<configentry name="Shut_Prio_FA_XSCF2">


</configentry>

<configentry name="Shut_Prio_FA_XSCF">


</configentry>

<configentry name="Shut_Prio_FA_RPS">


</configentry>

<configentry name="Shut_Prio_FA_VM">


</configentry>

<configentry name="Shut_Prio_FA_HV">


</configentry>

<configentry name="Shut_Prio_FA_GENERIC">


</configentry>

<configentry name="Shut_Prio_RESERVED_1">


</configentry>



</configentry>



</configentry>



</configentry>



</configentry>

<configentry name="Shut_Prio_WINSTD">

Parameter Reference



</configentry>

<configentry name="Shut_Prio_WIN_RESERVED_1">


</configentry>



</configentry>



</configentry>



</configentry>



</configentry>

<configentry name="Shut_UseIfIP_Blade">


</configentry>

<configentry name="Shut_UseIfIP_IPMI">


</configentry>

<configentry name="Shut_UseIfIP_IPMIPOWER">


</configentry>

<configentry name="Shut_UseIfIP_RSB">


</configentry>

<configentry name="Shut_UseIfIP_XSCF">


</configentry>

<configentry name="Shut_UseIfIP_RPS">


</configentry>

<configentry name="Shut_UseIfIP_SCON">


</configentry>

<configentry name="Shut_UseIfIP_XSCF2">


</configentry>

<configentry name="Shut_UseIfIP_ALOM">


</configentry>

<configentry name="Shut_UseIfIP_FA_BLADE">


</configentry>

<configentry name="Shut_UseIfIP_FA_IPMI">


</configentry>

<configentry name="Shut_UseIfIP_FA_XSCF2">


</configentry>

<configentry name="Shut_UseIfIP_FA_XSCF">


</configentry>

<configentry name="Shut_UseIfIP_FA_RPS">


</configentry>

<configentry name="Shut_UseIfIP_FA_VM">


</configentry>

<configentry name="Shut_UseIfIP_FA_HV">


</configentry>

<configentry name="Shut_UseIfIP_FA_GENERIC">


</configentry>

<configentry name="Shut_UseIfIP_RESERVED_1">


</configentry>


Parameter Reference



</configentry>



</configentry>



</configentry>



</configentry>

<configentry name="Shut_UseIfIP_WINSTD">


</configentry>

<configentry name="Shut_UseIfIP_WIN_RESERVED_1">


</configentry>



</configentry>



</configentry>



</configentry>



</configentry>

</configsection>



<configentry name="PrePoffHookPath">

<value

type="String">/opt/myAMC/scripts/PrePoffHook.sh</value>

</configentry>



<configentry name="PostPoffHookPath">

<value

type="String">/opt/myAMC/scripts/PostPoffHook.sh</value>

</configentry>



<configentry name="IgnorePoffHookResult">


</configentry>



<configentry name="VMHostPoffCheckPath">

<value

type="String">/opt/myAMC/scripts/VMHostPoffCheck.sh</value>

</configentry>



<configentry name="VMHostPoffCheckTimeout">


</configentry>

<configsection name="Managementblades">




<!-- this is in comment, because it is only an example !!!

<configsection name="Mgmt_Blade_1">


<value type="String">vader</value>

</configentry>

Parameter Reference


</configsection>

-->




<configsection name="Mgmt_Blade_2">


<value type="String">yoda</value>

</configentry>

</configsection>

-->

</configsection>





<configentry name="Default_ShutdownMode">


</configentry>





<configentry name="Shut_Cycle">


</configentry>

<configentry name="Shut_SNMP_Timeout">


</configentry>

<configentry name="Shut_SNMP_Tries">


</configentry>

<configentry name="Shut_Exec_Timeout">


</configentry>

<configentry name="Shut_PowerOff_Try">


</configentry>

<configentry name="Shut_ReNew_validity_1">


</configentry>

<configentry name="Shut_ReNew_validity_0">


</configentry>

<configentry name="Shut_DiagTrapRepeat_valid">

Parameter Reference


</configentry>

<configentry name="Shut_DiagTrapRepeat_invalid">


</configentry>

<configentry name="IgnoreShutdownFailure">


</configentry>

<configsection name="BrutForceShutdown">







<configsection name="BFSD_Blade_1">


<value type="String">vader</value>

</configentry>


<value type="String">XSCF</value>

</configentry>

<configentry name="MacAdress">

<value

type="String">00c00d0032f7</value>

</configentry>


<value type="String">SOLARIS</value>

</configentry>


<value type="String">cycle</value>

</configentry>

<configentry name="IP_Adress">

<value

type="String">158.92.4.214</value>

</configentry>


<value type="String">console_1</value>

</configentry>


<value type="String">i686</value>

</configentry>



</configentry>

</configsection>

-->




<configsection name="BFSD_Blade_2">


<value type="String">yoda</value>

</configentry>


<value type="String">BLADE</value>

</configentry>

<configentry name="MacAdress">

<value

type="String">00c00d0032f7</value>

</configentry>

Parameter Reference



<value type="String">LINUX</value>

</configentry>


<value type="String">cycle</value>

</configentry>

<configentry name="IP_Adress">

<value

type="String">192.168.102.104</value>

</configentry>


<value type="String">console_1</value>

</configentry>


<value type="String">i686</value>

</configentry>



</configentry>

</configsection>

-->

</configsection>

</configsection>

<configsection name="Additional_Checks">



<configentry name="AddCheckCycleTime">


</configentry>



<configentry name="Lock_file">

<value

type="String">/opt/myAMC/vFF/vFF_${vFF}/log/AppAgt/CHK_lock_</value>

</configentry>

<configsection name="File_check">








<configsection name="File_monitor_alert">

<configentry name="Filename">

<value

type="String">/opt/myAMC/scripts/sap/monitor_alert</value>

</configentry>

<configentry name="permissions">

<value type="String">FRX</value>

</configentry>

</configsection>

Parameter Reference


<configsection name="File_ff_service.sh">

<configentry name="Filename">

<value

type="String">/opt/myAMC/scripts/sap/ff_service.sh</value>

</configentry>


<value type="String">FRX</value>

</configentry>

</configsection>

</configsection>

<configsection name="Mount_check">








<configsection name="Mount_1">

<configentry name="Mountpoint">

<value

type="String">/FlexFrame/myAMC</value>

</configentry>


<value type="String">RW</value>

</configentry>

</configsection>

</configsection>

<configsection name="Filer_check">









<configsection name="Filer_1">

<configentry name="IP_Hostname">

<value type="String">filer</value>

</configentry>

<configentry name="ffu">

<value type="String">xxx</value>

</configentry>

</configsection>

-->




<configsection name="Filer_2">

<configentry name="IP_Hostname">

<value type="String">filer2</value>

</configentry>

<configentry name="ffu">

<value type="String">xxx</value>

</configentry>

</configsection>

-->

</configsection>

</configsection>

Parameter Reference


<configsection name="SharedResource">



<configsection name="Directory">




<configsection name="Directory_1">

<configentry name="RemoteNetwork">

<value

type="String">\\myserver\myuser</value>

</configentry>

<configentry name="User">

<value type="String">myuser</value>

</configentry>

<configentry name="Password">

<value

type="String">blabla1234567890</value>

</configentry>

</configsection>

-->



</configsection>

</configsection>

</configsection>

</configuration>

Parameter Reference


7.5.13 FA FrameAgent

The FA FrameAgent uses the same type of config files as the FA AppAgent and the FA

CtrlAgent. The FA FrameAgent is a local installation on the Control Nodes. Therefore the

config files are located in a local pool structure on the Control Node.

Local file system: /opt/local/myAMC/vFF/vFF_<Pool>/config/*

Instead of shared file system: /opt/myAMC/vFF/vFF_<Pool>/config/*

The default parameter values e.g. timing parameters, or path values in the config files

may differ.

7.5.14 Adjustment of the FA WebGUI

Some of the Icons shown in the FA WebGUI can be adapted with individual icons.

To change the standard icons with individual icons, the file

/opt/myAMC/FA_WebGui/FAWebGui.properties has to be changed.

application.title

Program title, when the FA WebGUI ist started as application

(/opt/myAMC/FA_WebGui/webgui.sh).

icons.logo-topleft

Left picture on the top left of the GUI.

Default is the „Fujitsu“ logo.

icons.logo-topright

Right picture on the top of the gui. Default is the „myAMC“ logo.

icons.heading

Icon in the middle of the top line of the GUI.

Default is„FlexFrame Autonomous Management Center“ icon.

icons.node.<POOLNAME>.<AN>

icons.node.bygroup.<poolname>.<groupname>

icons.node.<AN>

Icon for the Application Node with the name <AN>.

icons.nodes.<POOLNAME>

Icon for the Application Node of the pool <POOLNAME>.

icons.node.bygroup.<groupname>

icons.node.byos.<operatingsystem>

linux or solaris

Parameter Reference


icons.nodes

node

icons.system.<POOLNAME>.<SID>

icons.system.<SID>

icons.systems.<POOLNAME>

icons.systems

7.5.15 FA Webservices and Hypervisor Monitor configuration

The server part of the FA WebGui has some configuration parameters.

The configuration file is /opt/myAMC/config/FA_WebGui.conf

The following parameters are used for the collection of the virtual nodes of a hostsystem:

builder.enabled.vmhost

Enable or disable VMHost-Checkers: true or false

builder.update-cycletime.vmhost

Cycletime (in seconds) for the monitoring of available ESX servers.

Parameter Reference


7.6 FA DomainManager

7.6.1 Common

myAMC.FA-DomainManager is a component of myAMC.FA Performance and Accounting

Solution.

myAMC.FA-DomainManager processing measuring data which are collected from my-

AMC.FA-Agents and Collectors as ASCII-Files (called Collet-Files).

Reading of Collet-Files

Profil control, generation of alerts

Storage in database

7.6.2 Configuration

7.6.2.1 Common Parameters

The DomainManager needs the config file DomainManager.xml in the directory

/opt/myAMC/FA_DomainManager/config/.

LogLevel

Possible values: -1=Off, 1=Critical, 2=Error, 3=Warning, 4=Normal, 5=Info,

6=Babble, 7=Debug.

ReaderRefreshTime

Cycletime in seconds to look for new collets (default: 60).

7.6.2.2 Database Connection

The parameters for the database connection are in the same config file:

DatabaseDSN

ODBC database name (DSN), the same as in the file /etc/odbc.ini

(default: AMC_PERF).

DatabaseUser

Database user name (default: myAMC).

DatabasePassword

(default: FlexFrame).

Parameter Reference


7.6.2.3 Accounting Configuration

The additional values for the billing have to be configured with the FA ConfigGUI. The GUI is started with the script /opt/myAMC/FA_ConfigGui/configgui.sh.

7.6.2.3.1 SAPS Value and Hyperthreading

The DomainManager uses an application node’s computing capacity (measured in SAPS

based on SAP Release 4.7 32bit) und Hyperthreading state to calculate used compute

time for each service. The SAPS value and Hyperthreading state are automatically de-

tected by FA AppAgent and reported to DomainManager via the performance collet.

These values can be overridden. Additionally, the SAPS value can be converted to other

SAP releases.

Pool and host names allow the use of wildcards (“*”) to specify a group of nodes with a

single configuration entry.

Parameter Reference


7.6.2.3.2 Release Information for SAP Systems

The calculation of performance data for SAP systems depends on the release version.

Since the information of release in the collection is not included, this can be individually

configured for each system here. The system-specific configuration can be made for

every SID or for all SIDs or e.g. for productive SIDs with use of wildcards like “*” or “P*”.

Together with SAPS value per release the SAPS base can be changed.

Parameter Reference


7.6.2.3.3 Pricing Information

Here the necessary price information can be configured for the calculation of billing data.

Parameter Reference


7.6.2.4 Performance Profile

The profile for the monitoring of the performance values are set with the FA ConfigGUI. The GUI is started with the script /opt/myAMC/FA_ConfigGui/configgui.sh.

7.6.2.4.1 Profile for File System Monitoring

The thresholds for the file system monitoring are configured with this dialog.

Parameter Reference


7.6.2.4.2 Profile for Directory Monitoring

The dialog for the directory thresholds is configured with this dialog.


8 BlackBoard

8.1 General

myAMC.FA offers a command interface via which a node, an instance or a complete

system can be started and stopped. Furthermore it enables a system to be placed in the

nowatch status, which results in FA monitoring being disabled for this period. The com-

mand interface is used both for manual intervention and also for operation via the WebIn-

terface.

8.2 Implementation

The BlackBoard is implemented as an ASCII file which is secured against manipulation.

This file is also used as a log for the commands triggered via the BlackBoard.

Each BlackBoard command is valid for a particular period. After this the command is

discarded.

Each BlackBoard command has a mechanism which secures it against being modified

with an editor.

Command syntax:

Each command is represented by a line in the ASCII file blackboard.txt. This

line is formed of Variable=Value tuples. The following variables are available:

TimeStamp

Time the entry was made in the format “DD-MM-YY HH:MM:SS“

TimeLong

Timestamp of the entry (seconds since 1970)

SRC-ID

Source identifier (sender ID: "AppAgent", "CtrlAgent", "WebGui", "Extern" etc.)

SRC-Hostname

Host name of the sender

CMD-ID

Command identifier: String which identifies the command.

vFF

Name of the virtual FlexFrame for which this command applies.

Group

Name of the group for which this command applies.

Service

Name of the service for which this command applies.

BlackBoard


SID

SID of the SAP system for which this command applies.

Inst-Nr

Instance number for which this command applies.

Node

Host name of the node for which this command applies.

Value

Value for specific commands.

Validity

Validity period for this command in seconds (examples: "0", "180", etc.).

Key

String which protects the command against manipulation and entries made with

an editor.

Example:

TimeStamp=31-12-2003 15:00:10;

TimeLong=1234567899;

SRC-ID=myAMC.FA_WebInterface;

SRC-Hostname=vader;

CMD-ID=Service_Start;

vFF=Bayer1;

Group=Produktiv;

Service=SRV_APP;

SID=P*;

Inst-Nr=;

Node=;

Value=;

Validity=240;

Key=65h57709845kjjhjkh465480

Individual command syntax fields can be empty. These are then not entered in the

BlackBoard file.

Individual command syntax fields can be filled with keywords (e.g. "*"). Wildcards

can also be used.

An empty field does not mean “*” !

If, for example, a command is to apply for all nodes, Node=* must be specified.

BlackBoard


The following SRC-IDs (source identifiers) are currently permitted:

myAMC.FA_AppAgent

myAMC.FA_CtrlAgent

myAMC.FA_WebInterface

myAMC.FA_BBTool

myAMC.FA_BBTool_1

myAMC.FA_BBTool_2

myAMC.FA_BBTool_3

myAMC.FA_BBTool_4

myAMC.FA_BBTool_5

No other SRC-IDs are accepted.

BlackBoard


8.3 Generating BlackBoard Commands

8.3.1 WebInterface

Commands can be issued to the BlackBoard from the myAMC.FA WebInterface if in-

teraction is enabled or if the user’s rights permit this.

Here the variables required are requested by the users, if necessary, and the command

is written to the BlackBoard.

Over the Webinterface only the commands permitted for the relevant element (pool,

group, node, service) are offered.

In future enhancements it will also be possible to store security prompts and password

queries here to provide the greatest degree of security against user errors.

The actual entry in the BlackBoard file is made using the BBTool.sh script.

Hint: This interaction with WebInterface will be not supported by FlexFrame.

Example: Usage of WebInterface to generate BlackBoard commands:

With “right-click” to a node/service in the tree the context menu will be shown. There

services can be started, stopped or set to Watch/NoWatch.

Then the “Execute service action”-menu will appear, where depending on the selected

action different choices may be done.

BlackBoard


Next press „Execute“ to write the action in the BlackBoard. The further execution will be

done by the FA-Agents (after some delay).

8.3.2 Interactive

BlackBoard commands can be entered manually in the BlackBoard file using the

BBTool.sh and BBT_dialog.sh scripts.

The scripts and programs only provide a limited plausibility check.

You should therefore prefereably use the myAMC.FA WebInterface.

8.4 Commands

Command Parameters

(wildcards permitted)

Parameters

(no wildcards)

Value ?

Service_Start Node, vFF Group, Service,

SID, Inst-Nr

(with APP)

BlackBoard


Command Parameters

(wildcards permitted)

Parameters

(no wildcards)

Value ?

Service_Start_as_SwGet

(for internal use only)

Node, vFF Group, Service,

SID, Inst-Nr

(with APP)

Service_Stop Node, vFF, Group,

Service, SID, Inst-Nr

Service_Restart Node, vFF, Group,


Service_SetPrio Node, vFF, Group,


Value Priority

Service_Watch Node, vFF, Group,


Service_Nowatch Node, vFF, Group,


Service_ReactionON Node, vFF, Group,


Service_ReactionOFF Node, vFF, Group,


Service_TrapsendON Node, vFF, Group,


Service_TrapsendOFF Node, vFF, Group,


Node_Reboot Node, vFF

Node_Shutdown Node, vFF

Node_Switchover Node, vFF

Free command

(only autonomous test and

support)

Node, vFF Value Command


9 FlexFrame Autonomous Agent Traps

FlexFrame Autonomous Agents can be easily linked into Enterprise Management scenar-

ios. They supply an SNMP trap for all major status changes and for all reactions which

are implemented.

9.1 General

Each trap supplies all the important information on the time, physical and virtual identifi-

cation of the trap sender, and the severity, through to a meaningful message text in short

or long form, enabling it to be used directly for display in Enterprise Event Management

Systems or as a brief info, mail or SMS.

The contents of the traps of the FlexFrame Agents V1.0 and V2.0 are in principle identi-

cal. Trap attributes which did not exist in Version 1.0, e.g. pool name and group name,

are not sent.

The format of the FlexFrame Autonomy traps is such that they can be analyzed and fur-

ther processed in Enterprise Management Systems using filter modules. All important

attributes which concern FlexFrame Autonomy can be found in the variable bindings of a

FlexFrame Autonomy trap.

Enterprise OID 1.3.6.1.4.1.231.694.27,

Major Trap ID 6

Minor Trap ID 2

All messages contain a severity code. The following section describes the different sever-

ity levels:

Normal

Messages which indicate the transition to a normal state are sent with severity nor-

mal.

Warning

Messages regarding problems which are detected by the FA Agents are sent with

severity warning.

Critical

Messages regarding problems which lead to a reaction (restart, reboot, switchover)

are sent with severity critical.

Emergency

Messages regarding severe problems or situations are sent with severity emergen-

cy. This includes messages regarding MultiNodeFailures, which is a new feature as

of Version 3.0A10 of the FA Agents.


9.2 Structur of FlexFrame Autonomy Traps

All traps are sent with the Enterprise OID 1.3.6.1.4.1.231.694.27, plus the Major Trap ID 6 and the Minor Trap ID 2.

General trap format (all variable bindings have the prefix 1.3.6.1.4.1.231.694).

FlexFrame Autonomous Agent Traps


Example VB Description Comment

1 10 Trap version

27 20 Application ID (One of the two fields is always configurable)

myAMC.FA 21 Application name

1049117190 30 Timestamp (unixtime)

2003-04-01 31 Date %Y-%m-%d

12:15:00 32 Time %H:%M:%S

110 Symbolic group name (if configured)

111 Group name (if configured)

o11 121 System ID

belana 131 Physical device name

158.92.4.215 132 Physical IP address

1 140 Type: Alarm (1), Event (2), Log (3) ,

Heartbeat (4)

FA:ServiceUp

Log

152 Category (see VB 210) FA:<unique text as trap-id>

Log

00 161 Instance number

cio11_o11_00 162 Instance name <virtual_servername>_<sid>_<inst_no>



Example VB Description Comment

250 170 Severity (see table)

Cust_1 190 Pool name (vFF) Name of pool to which the host belongs

BSP_GR_1 191 Group name Group name of the service

2 192 Priority Priority of the service

SRV_DBORA 200 Service type name Unique service type name

The following names are possible: SRV_DBORA, SRV_DBSAP,

SRV_CI, SRV_APP, SRV_SCS, SRV_ASCS, SRV_JC, SRV_J

message server 201 Subservice type

Down 202 Status

cio11 203 Virtual host name (server LAN)

00 204 Service ID In the first version it corresponds to the instance number (VB 161)

myAMC_FA_AppAgent 205 Sender Process name : myAMC_FA_{App|Ctrl}Agent

db 206 Service type display name Display name of service type

123 210 Message ID (see VB 152) Unique trap number (internal ID)

message server is down 500 Short message Foramatted short message text

message server of service ci

(O11 00) on node belana is down

501 Long message Foramatted long message text



9.3 Overview of the FlexFrame Autonomy SNMP Traps

The following tables provide an overview of all defined traps of the FlexFrame Agents. Trap version V1 and V2.

These lists can be modified and extended at any time as a result of change requests.

A minus sign in a field means that this VB does not exist in the trap.

Table 1

VB 210

(as of

V2.0)

Internal ID see VB 210

(as of V2.0)

Short Message

VB 500

Long Message

VB 501

1 CtrlAgtUp has started <sender> on node <phys. server> has started

2 CtrlAgtDown has shut down <sender> on node <phys. server> has shut down

3 CtrlAgtSwitchOver switch over switching over service <service> <SID> <ID> node

<phys. server>

4 CtrlAgtSwitchOverFailed switch over failed switching over of service <service> <SID> <ID> from node

<phys. server> failed

5 CtrlAgtPoffHost_ok power off ok ‘<hostname>’ power off ok: <specific message>

6 CtrlAgtPoffHost_failed power off failed ‘<hostname>’ power off failed: <specific message>

7 CtrlAgtPoffHost_IF_ERR power off not done ‘<host-

name>’ power off failed: <specific message>

8 CtrlAgtPoffSwitchOffNetInterfacesOk switched off network interfaces switched off network interfaces



Table 1

VB 210

(as of

V2.0)


(as of V2.0)

Short Message

VB 500

Long Message

VB 501

9 CtrlAgtPoffSwitchOffNetInterfacesFailed failed to switch off network

interfaces failed to switch off network interfaces

10 CtrlAgentPoffNotDone Power off not done

'<hostname>’

Power off not done '<hostname>’ , because node may be

available. Administrator, check availability of node.

11 CtrlAgentPoffHhk_PING_ok ext. SwitchOver-Check

'<hostname>’: ping ok

ext. SwitchOver-Check '<hostname>’: ping ok <specific

message>

12 CtrlAgentPoffHhk_PING_fail ext. SwitchOver-Check

'<hostname>’: ping failed

ext. SwitchOver-Check '<hostname>’: ping failed <specific

message>

13 CtrlAgentPoffHhk_SSH_ok ext. SwitchOver-Check

'<hostname>’: ssh ok

ext. SwitchOver-Check '<hostname>’: ssh ok <specific

message>

14 CtrlAgentPoffHhk_SSH_fail ext. SwitchOver-Check

'<hostname>’: ssh failed

ext. SwitchOver-Check '<hostname>’: ssh failed <specific

message>

15 CtrlAgentMultiFailure_ShortTime MultiNodeFailure: ShortTime. MultiNodeFailure: ShortTime. <specific message>

16 CtrlAgentMultiFailure_LongTime MultiNodeFailure: LongTime. MultiNodeFailure: LongTime. <specific message>

17 CtrlAgentMultiFailure_AllNodes MultiNodeFailure: AllNodes. MultiNodeFailure: AllNodes. <specific message>

18 CtrlAgentNodeLiveMessageFailed Node live message failed

‘<hostname>’. Node live message failed ‘<hostname>’. <specific message>



Table 1

VB 210

(as of

V2.0)


(as of V2.0)

Short Message

VB 500

Long Message

VB 501

19 CtrlAgentNodeNotRechable_SwitchOver_decided Node not reachable, Switch-

Over decided ‘<hostname>’.

Node not reachable, SwitchOver decided ‘<hostname>’.

<specific message>

20 CtrlAgentNoSparePool_defined No spare pool defined. No spare pool defined. <specific message>

21 CtrlAgentNoSpare_to_TakeOver No spare for TakeOver. No spare for TakeOver. <specific message>

22 CtrlAgent_SPCT_SpareList_AddTo Add to list of spare nodes:

‘<hostname>’.

Add to list of spare nodes: ‘<hostname>’. <specific mes-

sage>

23 CtrlAgent_SPCT_SpareList_RemoveFrom Remove from list of spare

nodes: ‘<hostname>’.

Remove from list of spare nodes: ‘<hostname>’. <specific

message>

24 CtrlAgent_SPCT_RequestNotSuccessful Request not successful for

node: ‘<hostname>’.

Request not successful for node: ‘<hostname>’. <specific

message>

25 CtrlAgent_SPCT_MoveNodeSuccessful fa_move_node.sh successful

for node: ‘<hostname>’.

fa_move_node.sh successful for node: ‘<hostname>’. <spe-

cific message>

26 CtrlAgent_SPCT_MoveNodeNotSuccessful

fa_move_node.sh not suc-

cessful for node: ‘<host-

name>’.

fa_move_node.sh not successful for node: ‘<hostname>’.

<specific message>

27 CtrlAgent_VM_LicenseToKillServer Get the license to kill the VM-

Host. VM: ‘<hostname>’.

Get the license to kill the VM-Host. VM: ‘<hostname>’.

<specific message>



Table 1

VB 210

(as of

V2.0)


(as of V2.0)

Short Message

VB 500

Long Message

VB 501

28 CtrlAgent_VM_NOLicenseToKillServer Get no license to kill the VM-

Host. VM: ‘<hostname>’.

Get no license to kill the VM-Host. VM: ‘<hostname>’. <spe-

cific message>

29 CtrlAgent_VM_Message VM-Host check gives a mes-

sage. VM: ‘<hostname>’.

VM-Host check gives a message. VM: ‘<hostname>’. <spe-

cific message>

30 CtrlAgtPoffHost_SD Valuid shutdown valid ‘<hostname>’ shutdown valid: <specific message>

31 CtrlAgtPoffHost_SD_Invalid shutdown invalid ‘<hostname>’ shutdown invalid: <specific message>

100 AgentUp has <state> <sender> on node <phys. server> has <state>

101 AgentDown has <state> <sender> on node <phys. server> has <state>

110 NodeShutDown (as of V2.0) node <state> node <phys. server> has <state>

111 NodeRebooting (as of V2.0) node <state> node <phys. server> has <state>

112 NodeRebootStart (as of V2.0) node <state> node <phys. server> has <state>

113 NodeSwitchOver (as of V2.0) node <state> node <phys. server> has <state>

115 NodeTakeOverFailed (as of V2.0) node <state> node <phys. server> has <state>

200 ServiceStarting is <state> service <service> on node <phys. server> is <state>

201 ServiceUp has <state> service <service> on node <phys. server> has <state>



Table 1

VB 210

(as of

V2.0)


(as of V2.0)

Short Message

VB 500

Long Message

VB 501

202 ServiceStartFailed has <state> service <service> on node <phys. server> has <state>

203 ServiceStopping is <state> service <service> on node <phys. server> is <state>

204 ServiceDown has <state> service <service> on node <phys. server> has <state>

205 ServiceStopFailed has <state> service <service> on node <phys. server> has <state>

206 ServiceFailed <state> service <service> on node <phys. server> <state>

207 ServiceRestart (as of V2.0) is <state> service <service> on node <phys. server> is <state>

208 ServiceRestartFailed <state> service <service> on node <phys. server> <state>

209 ServiceWatch (as of V2.0) <state> watching service <service> on node <phys. server>

210 ServiceNowatch (as of V2.0) <state> No longer watching service <service> on node <phys. serv-

er>

211 ServiceReboot (as of V2.0) Is <state> service <service> on node <phys. server> is <state>

212 ServiceRebootStart (as of V2.0) Is <state> service <service> on node <phys. server> is <state>

213 ServiceSwitchOver (as of V2.0) Is <state> service <service> on node <phys. server> is <state>

214 ServiceSwitchOverStart (as of V2.0) Is <state> service <service> on node <phys. server> is <state>



Table 1

VB 210

(as of

V2.0)


(as of V2.0)

Short Message

VB 500

Long Message

VB 501

215 ServiceFlagPhase (as of V8.0) Is <state> service <service> on node <phys. server> is <state>

300 (1)

SubServiceDown is <state> saposcol on node <phys. server> is <state>

300 (2)

<ServiceSubType> of service <service> on node

<phys. server> is <state>

301 SubServiceFailed <state> <ServiceSubType> of service <service> on node

<phys. server> <state>

302 SubServiceDeleted <state> <state> <ServiceSubType> for service <service> on node

<phys. server>

400 CHK_Filers Filer <state> <specific message>

401 CHK_Files File <state> <specific message>

402 CHK_Lock Lock <state> <specific message>

403 CHK_Mounts Mount <state> <specific message>

404 CHK_SRDF_NAS_FH_ErrCode SRDF Check ErrCode <specific message>

405 CHK_SRDF_NAS_FH_SRDFFailed SRDF failed <specific message>

406 CHK_SRDF_NAS_FH_AdminCall SRDF Admin intervention

necessary <specific message>



Table 1

VB 210

(as of

V2.0)


(as of V2.0)

Short Message

VB 500

Long Message

VB 501

410 CHK_TimeSync_JumpDetected

(for future use) Time jump detected <specific message>

411 CHK_TimeSync_DiffDetected

(for future use) Time difference detected <specific message>

450 PERF_SpaceUtilFS_VExc_MaxUtilization Perf Filesystem 'MaxUtiliza-

tion' exceeded <specific message>

451 PERF_SpaceUtilFS_VExc_MaxGrowthRate Perf Filesystem

'MaxGrowthRate' exceeded <specific message>

500 BB_get node <state> <specific message>

501 BB_DoIt node <state> <specific message>

502 BB_Error node <state> <specific message>

503 BB_NoMatch node <state> <specific message>



Table 2

VB 210

(as of V2.0) Service VB 200

Service Subtype

VB 201

ID

VB 204

Logical Server Name in

ServerLAN (SL) VB 203

SID

VB 121

Long Instance

No. VB 161

1 – 2 - - - - - -

3

SRV_DBORASRV_DBSAP, SRV_LC

-

- SL <SID> -

SRV_CI, SRV_JC, SRV_SCS,

SRV_ASCS, SRV_APP, SRV_J <ID> SL <SID> <Inst No>

SRV_ERS <ID> - <SID> <Inst No>

4

SRV_DBORASRV_DBSAP, SRC_LC

-

- SL <SID> -

SRV_CI, SRV_JC, SRV_SCS, RV_ASCS,

SRV_APP, SRV_J, SRV_ERS <ID> SL <SID> <Inst No>


5 – 10,

18 – 19

22 – 31

<hostname> - - - - -

1-2,

11-17, 20 -

21

100 – 101,

110 – 113,

115

- - - - - -



Table 2

VB 210


Service Subtype

VB 201

ID

VB 204



SID

VB 121

Long Instance

No. VB 161

200 – 215


-

- SL <SID> -




300 (1)

- saposcol - SL - -

300 (2)

SRV_DBSAP, SRV_LC vserver, kernel

- SL - - SRV_DBORA

ora_dbw, ora_lgwr,

ora_ckpt, ora_smon,

ora_pmon, tnslsnr,

ora_mman, ora_arc

SRV_ERS ers.sap <ID> - <SID> <Inst No>

SRV_CI ms.sap, dw.sap

<ID> SL <SID> <Inst No> SRV_JC jc.sap

SRV_SCS, SRV_ASCS ms.sap, en.sap

SRV_APP dw.sap



SRV_J jc.sap

Table 2

VB 210


Service Subtype

VB 201

ID

VB 204



SID

VB 121

Long Instance

No. VB 161

301


server-lan ping

- SL <SID> -




SRV_ASCS, SRV_APP, SRV_J client-lan ping <ID> SL <SID> <Inst No>

302


control file

- SL <SID> -




400 – 406,

410 – 411,

450 – 451,

500

- - - - - -

501 – 503

SRV_DBORASRV_DBSAP, SRV_CI,

SRV_JC, SRV_SCS, SRV_ASCS,

SRV_APP, SRV_J, SRV_ERS, SRV_LC

- <ID> SL <SID> <Inst No>



Table 3

VB 210


503 BB cmd no match myAMC_FA_[App]|[Frame]|[Ctrl]Agent 50, 150, 250


10 FA Troubleshooting

10.1 General

The FA Agents offer a large number of diagnostic options for detecting and diagnosing

problems on the FA Agents themselves or other components.

Problems concerning FA Agents can be assigned to one of the following categories:

FlexFrame installation and configuration errors

Parameter errors

Configuration errors

Detection, reaction errors, start, stop, maintenance errors

Power-shutdown errors

Typical consequences of installation and configuration errors are:

FA Agents fail to start

Error messages during startup of FA Agents

10.2 Incident Diagnosis

Error: Mount points missing

Diagnosis:

In the case of mount points monitored by FA Autonomy, traps are sent to the central

trap consoles. With other mount points which are absolutely essential for the opera-

tion of the node concerned it can happen that the agents cannot be started as the di-

rectories required are not available.

Response:

Provide the mount points required with the appropriate mount options.

Error: Mount points without “File Locking”

Diagnosis:

The FA Agents log this situation both in the operating system’s Syslog and in special files (/opt/myAMC/vFF/log/log_syslog*).

Response:

Provide the mount points required with the appropriate mount options (lock).

Error: Rights for the directories/files are not sufficient

FA Troubleshooting


Diagnosis:

In the case of files monitored by FA Autonomy, traps are sent to the central trap con-

soles. With other directories/files which are absolutely essential for the operation of

the node concerned it can happen that the agents cannot be started as the directo-

ries/files required are not available.

Response:

Provide the directories/files with the required rights.

Error: Agents do not have the authorization to write to the

directories assigned

Diagnosis:

The FA Autonomy work and log files are not written.

Response:

Provide the directories/files with the required rights.

Error: Version incompatibility

FlexFrame installation and FlexFrame Autonomy installation are not directly compati-

ble. This can always be the case when older FlexFrame installations are updated

with new FlexFrame Autonomy Agents.

Diagnosis:

For diagnosis and troubleshooting, the mount points, the directory structure and the

access rights to the directories used by the agents must be checked.

Response:

Using the command line tool, check that the parameters used in the FA config files

are compatible with the version and syntactically correct.

Error: Pool assignment not found

A node is assigned to the wrong pool or to the default pool.

Diagnosis:

Display on the FA_WebGUI or in the agent’s start trap and display on an event con-

sole.

Response:

Check the LDAP configuration parameters, call the PGTool Pool.sh and check the

pool name returned.

Check the pool membership for each node.

Error: Group assignment is not correct

FA Troubleshooting


Diagnosis:


sole.

Response:

Check the group configuration in the group configuration file. Check the group mem-

bership for each node.

Error: Service priority not recognized

Diagnosis:


sole.

Response:

Check the configuration of the service class and service priority in the group configu-

ration file. Check the values for each node.

Error: Availability problem not rectified by autonomous reaction

Diagnosis:

Services are discontinued (possibly due to hardware fault) and are not made avai-

lable again by FlexFrame Autonomy.

Response:

Check whether nodes are available for taking over the services (Spare Nodes).

Check whether the FA Agents on the nodes involved have been started.

Error: Services do not start

Constant reboot

Permanent switchover

Diagnosis:

SAP services which are started do not enter run mode but are repreatedly restarted

or, if the problem escalates, the node is rebooted or an internal SwitchOver takes

place. Possible causes:

The MaxRestart time for the service is too short. This parameter can be adjusted in

the FA configuration.

The virtual interfaces cannot be reached.

There is a permanent problem which prevents a service being started (e.g. neces-

sary database recovery).

Response:

Stop the FA Agents to interrupt escalation of the reaction and check whether the ser-

vice can be started manually.

If the service cannot be started manually, this problem must be corrected by the ad-

FA Troubleshooting


ministrator.

If the service can be started manually, the time required for this must be matched to

the MaxStart time and MaxRestart time in the configuration and the configuration

must be adjusted, if necessary.

If the virtual interfaces cannot be reached from the Application Agent, the network

configuration must be checked.

Error: Service cannot be stopped

Diagnosis:

An active SAP service is repeatedly restarted after a manual stop command .

Response:

If the FlexFrame SAP scripts were not used for the manual stop command, this is the

cause. The FlexFrame SAP scripts must be used.

If the FlexFrame SAP scripts were used, the Monitor Alert Script might not be availa-

ble or does not have the required rights.

However, it is also possible that the Monitor Alert Time and CycleTime are configured

incorrectly. The agents’ CycleTime is too long in relation to the Monitor Alert Time.

The Monitor Alert Time must be at least 3 * the CycleTime.

Error: Maintenance activities are interrupted by autonomous

reactions

Diagnosis:

Unwanted autonomous reactions during maintenance.

Response:

Set NoWatch for the service concerned or stop the Application Agents for the node

concerned and restart them after maintenance has been completed.

Error: Incorrect display on the FA_WebGUI

Diagnosis:

The state checked manually does not match the display.

Response:

Check whether the Application Agents concerned, the Control Agents concerned and

the web server are running properly for the WebInterface.

FA Troubleshooting


10.3 FA Work and Log Files

10.3.1 General

The FlexFrame Autonomy Agents write detailed log files.

The log files of the FlexFrame Autonomy Agents can be used for the diagnosis.

The functions of the FA Agents are documented in their own files. These files may not be

modified manually as this can impair fault-free operation of the FA Agents or lead to erro-

neous reactions.



they reevaluate the situation from the current viewpoint without any previous knowledge.

10.3.2 Overview, Principal Directories, Files

Base directory: /opt/myAMC/

Base directory: /opt/local/myAMC/

Version numbers: In the table below, V<v>K<r> corresponds to

V<version number>K<revision_number>.

Subdirectories Content

./scripts Scripts for various tasks

./scripts/sap Link to the FlexFrame scripts

./scripts/PowerMng Scripts for the power managemnet

blades.

./scripts/ShutDown_Node Scripts to shutdown a node.

./scripts/fa_list_services.sh Script to list all FlexFrame service states.

./scripts/allnodes Script to execute a command at all nodes

of a pool.

./config General configuration data

FA Troubleshooting



./config/FA_WebGui.conf General settings for the WebGUI (directo-

ries, cycle times, database settings)

./config/amc-users.xml User management

./FA_AppAgent Installation path of myAMC. FA_AppAgent

and of diverse scripts.

./FA_AppAgent/myAMC.FA_AppAgent Start/Stop scripts AppAgent

./FA_AppAgent/PGTool_Pool.sh Determination of pool membership

./FA_AppAgent/PGTool_Version.sh Determination of pool version

./FA_AppAgent/PVget.sh Determination of the SAPS number of a

node

./FA_ AppAgent/BBTool.sh BlackBoard control

./FA_ AppAgent/BBT_dialog.sh BlackBoard dialog mode control

./FA_AppAgent/bin_Linux_SLES9_<VxxKyy>.

/FA_AppAgent/lib_Linux_SLES9_<VxxKyy>

Binaries and libraries for each version.

./FA_AppAgent/config myAMC.FA_AppAgent-specific configura-

tion data

./FA_AppAgent/log empty

./FA_CtrlAgent Installation path of myAMC. FA_CtrlAgent

and of scripts

./FA_CtrlAgent/myAMC.FA_AppAgent Start/StopScripts CtrlAgent

./FA_CtrlAgent/PGTool_Pool.sh Determination of pool membership

./FA_CtrlAgent/PGTool_Version.sh Determination of pool version

./FA_CtrlAgent/PVget.sh Determination of the SAPS number of a

node

./FA_CtrlAgent/BBTool.sh BlackBoard control

./FA_CtrlAgent/BBT_dialog.sh BlackBoard dialog mode control

./FA_CtrlAgent/bin_Linux_SLES9_<VxxKyy>

./FA_CtrlAgent/lib_Linux_SLES9_<VxxKyy>


FA Troubleshooting



./FA_CtrlAgent/config myAMC.FA_CtrlAgent-specific configura-

tion data

./FA_CtrlAgent/log empty

./FA_FrameAgent Installation path of myAMC.

FA_FrameAgent and of diverse scripts.

./FA_FrameAgent/myAMC.FA_FrameAgent Start/Stop scripts FrameAgent

./FA_FrameAgent/PGTool_Pool.sh Determination of pool membership

./FA_FrameAgent/PGTool_Version.sh Determination of pool version

./FA_FrameAgent/PVget.sh Determination of the SAPS number of a

node

./FA_ FrameAgent/BBTool.sh BlackBoard control

./FA_ FrameAgent/BBT_dialog.sh BlackBoard dialog mode control

./FA_FrameAgent/bin_Linux_SLES9_<VxxKyy

>./FA_FrameAgent/lib_Linux_SLES9_<VxxKy

y>


./FA_FrameAgent/config myAMC.FA_FrameAgent-specific configu-

ration data

./FA_FrameAgent/log empty

./FA_ConfigGui Installation path of myAMC.

FA_ConfigGui.

./FA_DomainManager Installation path of myAMC.

FA_DomainManager.

./FA_LogAgent Installation path of myAMC.

FA_LogAgent.

FA Troubleshooting



./FA_Messenger Installation path of myAMC.

FA_Messenger.

./FA_ Messenger/log/Robot.txt Protocolfile of received messages.

./FA_WebGui Installation path of myAMC. FA_WebGui.

./vFF Pool-specific (vFF) data

./vFF/SpareRequest Request directory for spares from spare

pool

./vFF/log Pool-specific log files

./vFF/log/myAMC_FA_Pools.xml

./vFF/log/myAMC_FA_Pools-default.xml

Pools configuration file and its default

version (this is used as LDAP cache)

./vFF/Common/.vFF_template.<VxxKxx> Template of pool-specific data for each

version

./vFF/Common/.vFF_template.<VxxKxx>/

config

Configuration


config/TrapTargets.xml

Trap targets


config/myAMC_FA.xml


config/myAMC_FA-default.xml

myAMC.FA configuration and delivery

status (default).


config/myAMC_FA_Checks.xml


config/myAMC_FA_Checks-default.xml

Definition of „additional Checks“ and

delivery status (default).

FA Troubleshooting




config/myAMC_FA_GUI.xml


config/myAMC_FA_GUI-default.xml

myAMC.FA GUI configuration and deliv-

ery status (default).


config/myAMC_FA_Groups.xml


config/myAMC_FA_Groups-default.xml

myAMC.FA groups configuration and

delivery status (default).

This is used as LDAP cache


config/myAMC_FA_Hosts.xml


config/myAMC_FA_Hosts-default.xml

Host configuration from LDAP and deliv-

ery status (default).

This is used as LDAP cache


config/myAMC_FA_SD_Sec.xml


config/myAMC_FA_SD_Sec-default.xml

myAMC.FA shutdown security configura-

tion and delivery status (default).


config/myAMC_FA_Rules.xml


config/myAMC_FA_Rules-default.xml

myAMC.FA service rules and delivery

status (default).


config/myAMC_FA_SelectionRules.xml


config/myAMC_FA_SelectionRules-

default.xml

Definition of „SelectionRules“ for „Pool

independent spare node“ and delivery

status (default).

./vFF/Common/.vFF_template.<VxxKxx>/log


/AppAgt


/CtlrAgt


/FrameAgt

Logfiles for each pool of:

myAMC.FA_AppAgent

myAMC.FA_CtrlAgent

myAMC.FA_FrameAgent


data


data/FA

Work files


data/FA/livelist

Live list

livelist.log

FA Troubleshooting




data/FA/xmlrepository

XML repository for the web interface

livelist.

xmlServices_<nodename>.xml


data/FA/servicelists

Service lists

Services_<nodename>.lst


data/FA/servicelogs

Service logs (history)

Services_<nodename>.log


data/FA/reboot

Reboot files

Reboot_<nodename>.lst

Persistent FA-Info

Persist_FA_Info.txt


data/FA/switchover

SwitchOver files

SwitchOver_<nodename>.lst


data/FA/blackboard

BlackBoard

blackboard.txt


data/FA/performance

Measured performance data

./vFF/Common/.vFF_Master Pool-Master with the same directories and

files as in template

./vFF/vFF_Cust_1

./vFF/vFF_Cust_1/config

./vFF/vFF_Cust_1/log/…

./vFF/vFF_Cust_1/data

./vFF/vFF_Cust_1/data/FA/….

Pool-specific data for pool “Cust_1”

(example).

See above for the description of the sub-

directories and files.

./vFF/vFF_Cust_2

./vFF/vFF_Cust_2/config

./vFF/vFF_Cust_2/log/…

./vFF/vFF_Cust_2/data

./vFF/vFF_Cust_2/data/FA/….

Pool-specific data for pool “Cust_2”

(example).

See above for the description of the sub-

directories and files.

FA Troubleshooting


10.3.3 Collecting Diagnostic Information for Support Assis-tance

If support is needed, there is special data needed in the FlexFrame Support. This infor-

mation is required to analyze problems with FlexFrame and the Autonomous Agents.

Error description, as precise as possible

What is the problem or error? On which pool/node/service does it occur?

Version of the FA Agents installed

Run rpm -qa | grep myAMC on the Control Node.

Configuration, work and log files of the FA Agents

The following script creates an archive with the desired information:

/opt/myAMC/FA_CtrlAgent/SAVE_FA_files_for_diag.sh

This script must be invoked on the controlnode!

cd /opt/myAMC/FA_CtrlAgent

./SAVE_FA_files_for_diag.sh

Usage


./SAVE_FA_files_for_diag.sh –h usage: SAVE_FA_files_for_diag.sh <args>

-h (show this help)

-C (save config only)

-c "<customer name>" specify customer name

-i "<incident number>" specify incident number

-d "<problem description>" specify problem description

-P "<pools>" specify pools

-p <y Y n N> include performance files [Y/N]

-g <y Y n N> compress output file [Y/N]

-D LogFiles Delete log files

-D PerfFiles Delete performance files

-D LogAndPerfFiles Delete log and performance files

Please ensure, that this script will be started at the active Ctrl-Node (CtrlAgent

is active).

FA Troubleshooting



changed manually as this can impair error-free operation of the FA Agents or result in

incorrect reactions.



they re-evaluate the situation from the current viewpoint without any previous knowledge.

FA Troubleshooting


10.3.4 Selected Files

The write cycle for the entries (with the exception of reboot, switchover and BlackBoard)

and the storage location of the files described in the following are defined using a param-

eter in the configuration file myAMC_FA.xml.

10.3.4.1 Livelist

Each FA AppAgent regularly enters itself in this list. Through these entries the my-

AMC.FA_CtrlAgent recognizes whether the various myAMC.FA_AppAgents are available

and functioning without error.

10.3.4.2 Services List

This file (testament) exists for each FA AppAgent on a node-specific basis. In it the agent

logs the services which it has detected using its detectors plus their current status. A

service-related status is logged in this file. The contents are updated after each detector

cycle.

10.3.4.3 Services Log

The contents of this file are identical to those of the Services-List file, with the differ-

ence that the history is contained in this file. This enables status changes and reaction

decisions to be detected and replicated.

10.3.4.4 Reboot

The contents of this file are identical to those of the Services-List file. The file serves

as an information storage when a reboot takes place. It is written only for the autonomous

reaction reboot and is deleted again after the reboot has been completed and the ser-

vices have been started up.

10.3.4.5 Switchover

The contents of this file are identical to those of the Services-List file. The file serves

as information storage (testament) when a switchover takes place. It is written only for the

autonomous reaction switchover and is deleted again after the services have been taken

over.

10.3.4.6 XML Repository

In terms of contents the files in the XML Repository are the same as those in the Livelist

and Services List. By contrast, the contents are written in XML notation and can thus be

visualized directly with the associated FA WebInterface.

The write cycle for the entries and the storage location of the file are defined using a

parameter in the configuration file myAMC_FA.xml.

FA Troubleshooting


10.3.4.7 BlackBoard

The BlackBoard is an input interface for the FA Agents. Commands can be entered here

which are executed by the FA AppAgents. The commands have a specific validity period

and are secured against manipulation. The file is written manually using a tool which

guarantees, among other things, protection against manipulation.

10.3.5 FA Autonomy Diagnostic Tool

Manual diagnosis of the log files can be very time-consuming. The Fujitsu support organ-

ization works with specialized diagnostic tools which can analyze even large quantities of

data very quickly and efficiently. This service can be utilized when required and if a corre-

sponding service agreement exists.

To utilize this service, either individual log files or the entire virtual FA directory of a pool

can be sent to the support department, e.g. as a compressed and protected zip archive.

FA Troubleshooting


10.3.6 Collecting Diagnostic Information for Support Assis-tance

If support is needed, there is special data needed in the FlexFrame Support. This infor-

mation is required to analyze problems with FlexFrame and the Autonomous Agents.

Error description, as precise as possible

What is the problem or error? On which pool/node/service does it occur?

Version of the FA Agents installed

Run rpm -qa | grep myAMC on the Control Node.

Configuration, work and log files of the FA Agents

The following script creates an archive with the desired information:

/opt/myAMC/FA_CtrlAgent/SAVE_FA_files_for_diag.sh

This script must be invoked on the controlnode!


./SAVE_FA_files_for_diag.sh

Usage


./SAVE_FA_files_for_diag.sh –h usage: SAVE_FA_files_for_diag.sh <args>

-h (show this help)

-C (save config only)

-c "<customer name>" specify customer name

-i "<incident number>" specify incident number

-d "<problem description>" specify problem description

-P "<pools>" specify pools

-p <y Y n N> include performance files [Y/N]

-g <y Y n N> compress output file [Y/N]

-D LogFiles Delete log files

-D PerfFiles Delete performance files

-D LogAndPerfFiles Delete log and performance files

Please ensure, that this script will be started at the active Ctrl-Node (CtrlAgent

is active).

FA Troubleshooting



changed manually as this can impair error-free operation of the FA Agents or result in

incorrect reactions.



they re-evaluate the situation from the current viewpoint without any previous knowledge.


11 Abbreviations

ABAP Advanced Business Application Programming

ACI Adaptive Computing Infrastructure

ACPI Advanced Configuration and Power Interface

APM Advanced Power Management

APOLC Advanced Planner & Optimizer Life Cache

CCU Console Connection Unit

CIFS Common Internet File System

DART Data Access in Real Time

DHCP Dynamic Host Configuration Protocol

DIT Domain Information Tree

ERP Enterprise Resource Planning

ESF Enhanced System Facility

EULA End User License Agreement

FAA FlexFrame Autonomous Agent

FC Fiber Channel

FTP File Transfer Protocol

IP Internet Protocol

IPMP IP Multipathing

LAN Local Area Network

LDAP Lightweight Directory Access Protocol

LUN Logical Unit Number

LVM Landscape Virtualization Management

SAP system for monitoring and controlling SAP landscapes.

MAC Media Access Control

MINRA Minimal Read Ahead

NAS Network Attached Storage

NDMP Network Data Management Protocol

NFS Network File System

Abbreviations


NIC Network Interface Card

NVRAM Non-Volatile Random Access Memory

OBP Open Boot Prom

OLTP On-Line Transaction Processing

ONTAP Open Network Technology for Appliance Products

OSS Open Source Software

POST Power-On Self Test

PCL PRIMECLUSTER

PFS Production File System (on Celerra)

PW PRIMEPOWER

PXE Preboot Execution Environment

PY PRIMERGY

QA Quality Assurance

QS Quality of Service

RAID Redundant Array of Independent (or Inexpensive) Disks

RARP Reverse Address Resolution Protocol

RDBMS Relational Database Management System

RHEL Red Hat Enterprise Linux

RSB Remote Service Board

SCS System Console Software

SAP BW SAP Business Warehouse

SAPGUI SAP Graphical User Interface

SAPOSS SAP Online System Service

SID System Identifier

SLD System Landscape Directory

SLES SUSE Linux Enterprise Server

SMB Server Message Block

SMC System Management Console

SNMP Simple Network Management Protocol

Abbreviations


SPOC Single Point Of Control

TELNET Telecommunications Network

TFTP Trivial File Transfer Protocol

UDP User Datagram Protocol

UPS Uninterruptible Power Supply

VLAN Virtual Local Area Network

VTOC Virtual Table Of Contents

WAN Wide Area Network

WAS Web Application Server

WAFL Write Anywhere File Layout

XSCF Extended System Control Facility


12 Glossary

Advanced Business Application Programming

Proprietary programming language of SAP.

Advanced Power Management

Advanced Power Management defines a layer between the hardware and the operat-

ing system that effectively shields the programmer from hardware details.

Application Agent

A software program for monitoring and managing applications.

Application Node

A host for applications (e.g. SAP instances db, ci, agate, wgate, app etc.). This

definition includes Application Servers as well as Database Servers.

Automounter

The automounter is an NFS utility that automatically mounts directories on an NFS

client as they are needed, and unmounts them when they are no longer needed.

Autonomous Agent

Central system management and high availability software component of FlexFrame.

Blade

A special form factor for computer nodes.

BladeRunner

The working title for the solution part of SAP for FlexFrame.

BOOTPARAM

Boot time parameters of the kernel.

BRBACKUP

SAP backup and restore tools.

Celerra

NAS system of EMC.

Checkpoint Restore

On EMC Celerra a SnapSure feature that restores a PFS to a point in time using

checkpoint information. As a precaution, SnapSure automatically creates a new

checkpoint of the PFS before it performs the restore operation.

Client LAN

Virtual network segment within FlexFrame, used for client-server traffic.

Common Internet File System

A protocol for the sharing of file systems (same as SMB).

Glossary


Computing Node

From the SAP ACI perspective: A host that is used for applications.

Control Agent

A software program for monitoring and managing nodes within FlexFrame.

Control LAN

Virtual network segment within FlexFrame, used for system management traffic.

Control Node

A physical computer system, controlling and monitoring the entire FlexFrame land-

scape and running shared services in the rack (dhcp, tftp, ldap etc.).

Control Station

A Control Node in an SAP ACI environment.

DART

Operating system of Celerra data movers (Data Access in Real Time).

Dynamic Host Configuration Protocol

DHCP is a protocol for assigning dynamic IP addresses to devices on a network.

Dynamic Host Configuration Protocol server

A DHCP server provides configuration parameters specific to the DHCP client host,

required by the host to participate on the Internet.

EMC NAS

Network attached storage for file systems of EMC.

Enterprise Resource Planning

Enterprise Resource Planning systems are management information systems that in-

tegrate and automate many of the business practices associated with the operations

or production aspects of a company.

Ethernet

A Local Area Network which supports data transfer rates of 10 megabits per second.

Fiber Channel

Fiber Channel is a serial computer bus intended for connecting high-speed storage

devices to computers.

Filer

Network attached storage for file systems of NetApp.

FlexFrame

A joint project in which the main partners are SAP, Network Appliance, Intel and Fu-

jitsu.

FlexFrame® for SAP

®

FlexFrame® for SAP

® is a radically new architecture for SAP environments. It exploits

Glossary


the latest business-critical computing technology to deliver major cost savings for

SAP customers.

FlexFrame internal LAN Switch

Cisco network switches which are integral part of the FlexFrame for SAP hardware

configuration and which are automatically configured by the FlexFrame for SAP soft-

ware.

Gigabit Ethernet

A Local Area Network which supports data transfer rates of 1 gigabit (1,000 mega-

bits) per second.

Host name

The name of a node (assigned to an interface) that is resolved to a unique IP ad-

dress. One node can have multiple host names (cf. node name).

In SAP environments host names are currently limited to 13 alphanumeric characters

including the hyphen (“ - “). The first character must be a letter. In the SAP environ-

ment host names are case-sensitive.

Image

In the FlexFrame documentation, “Image” is used as a synonym for “Hard Disk Im-

age”.

Internet Protocol Address

A unique number used by computers to refer to each other when sending information

through networks using the Internet Protocol.

Landscape Virtualization Management (LVM)

SAP system for monitoring and controlling SAP landscapes

Lightweight Directory Access Protocol

Protocol for accessing on-line directory services.

Local Area Network

A computer network that spans a relatively small area. Most LANs are confined to a

single building or group of buildings. However, one LAN can be connected to other

LANs over any distance via telephone lines and radio waves. A system of LANs con-

nected in this way is called a Wide Area Network (WAN).

Local host name

The name of the node (physical computer); it can be displayed and set using the

command /bin/hostname.

Logical Unit Number

An address for a single (SCSI) disk drive.

MAC address

Device identifier number of a Network Interface Card. In full: "media access control

address".

Glossary


MaxDB

A relational database system from mySQL (formerly ADABAS and SAPDB).

Media Access Control address

An identifier for network devices, usually unique. The MAC address is stored physi-

cally on the device.

NAS system

Network Attached Storage of any vendor (in our context: EMC NAS or NetApp Filer).

NDMPcopy

NDMPcopy transfers data between Filers using the Network Data Management Pro-

tocol (NDMP).

Netboot

A boot procedure for computers where the operating system is provided via a net-

work instead of local disks.

Netweaver

SAP NetWeaver is the technical foundation of SAP solutions.

Network Appliance Filer

See “Filer”.

Network Attached Storage

A data storage device that is connected via a network to one or multiple computers.

Network File System

A network protocol for network-based storage access.

Network Interface Card

A hardware device that allows computer communication via networks.

Node

A physical computer system controlled by an OS.

Node name

The name of a physical node as returned by the command uname -n. Each node

name within a FlexFrame environment must be unique.

Non-Volatile Random Access Memory

A type of memory that retains its contents when the power is turned off.

On-Line Transaction Processing

Transaction processing via computer networks.

OpenLDAP

An Open Source LDAP Service Implementation.

Open Network Technology for Appliance Products

The operating system of Network Appliance Filers.

Glossary


Open Source Software

Software that is distributed free of charge under an open source license, such as the

GNU Public License.

Oracle RAC

A cluster database by Oracle Corporation.

Physical host

Name of a physical computer system (node).

Power-On Self Test

Part of a computer's boot process; automatic testing of diverse hardware compo-

nents.

Preboot Execution Environment

An environment that allows a computer to boot from a network resource without hav-

ing a local operating system installed.

PRIMECLUSTER

Fujitsu’s high-availability and clustering software.

PRIMEPOWER

Fujitsu's SPARC-based server product line.

PRIMERGY

Fujitsu's i386-based server product line.

Red Hat Enterprise Linux

Linux distribution by Red Hat, Inc., targeting business customers.

Reverse Address Resolution Protocol

A protocol allowing resolution of an IP address corresponding to a MAC address.

SAP Service

In FlexFrame: SAP Service and DB Services.

SAP service script

An administration script for starting and stopping an SAP application on a virtual host.

SAP Solution Manager

Service portal for the implementation, operation and optimization of an SAP solution.

SAPLogon

Front-end software for SAPGUI.

SAPRouter

Router for SAP services like SAPGUI or SAPTELNET.

SavVol

A Celerra volume to which SnapSure copies original point-in-time data blocks from

the PFS before the blocks are altered by a PFS transaction.

Glossary


Server

A physical host (hardware), same as node.

Service

A software program providing functions to clients.

Service type

The type of an application or service (db, ci, app, agate, wgate etc.).

Single Point of Control

In FlexFrame: One user interface to control a whole FlexFrame environment.

Storage LAN

A virtual LAN segment within a FlexFrame environment, carrying the traffic to NAS

systems.

SUSE Linux Enterprise Server

A Linux distribution by Novell, specializing in server installations.

Telecommunications Network

A terminal emulation program for TCP/IP networks such as the Internet.

Trivial File Transfer Protocol

A simple form of the File Transfer Protocol (FTP). TFTP uses the User Datagram

Protocol (UDP) and provides no security features. It is often used by servers to boot

diskless workstations, X-terminals, and routers.

TFTP server

A simple FTP implementation.

Virtual host

The name of the virtual host on which an application runs; it is assigned to a physical

node when an application is started.

Virtual Local Area Network

A VLAN is a logically segmented network mapped over physical hardware according

to the IEEE 802.1q standard.

Virtualization

Virtualization means the separation of hardware and processes. In a virtualized envi-

ronment (FlexFrame), a process can be moved between hardware nodes while stay-

ing transparent to the user and application.


13 Index

A

allnodes 158

autonomous functions 70

autonomous operation 127

autonomy

Adjustment of FA-WebGUI 306

application scenarios 97

collecting diagnostic information for

support assistance 353, 357

definition of generic services

parameters 267

directories, production and log

files 347

FA Frame Agent 306

general parameters 249

node-related parameters 263

parameters for the performance and

accounting option 251

path configuration 271

power shutdown 199

service-related parameters 265

shutdown configuration 273

user interactions 128

WebInterface 159

Autonomy 14

autonomy architecture 65

autonomy pools 70

autonomy reactions 98

Reboot 98

Restart 98

SwitchOver 98

Autonomy software 22

start scripts 23

autonomy traps 321

format 321, 323

overview 326

B

BlackBoard commands

command file 314

command list 318

interactive 318

via WebInterface 317

C

class creation rules 78

Client LAN 20

D

directories, production and log

files 149

Domain Manager 95

Data collection 95

Data retentiontime and

compression 96

DomainManager 47

Database connection 47

mySQL 48

PostgreSQL 48

Index


E

escalation with partner service 105

SCS/ASCS with ERS 105

service with SMD 107

F

FA Agent

deactivating and activating 20

installing 20

system and directory 20

version 20

FA Autonomy

diagnostic tool 356

FA components

configuration and log files 79

service types 80

systems 80

FA WebInterface 44

configuring 44

function 44

installing 44

starting and stopping 44

WebGui and database 45

mySQL 45

PostgreSQL 46

FA_AppAgents 19

FA_CtrlAgent 19

FA_DomainManager 19

fa_list_services.sh 157

FA_LogAgent 19

FA_Messenger 19

FA_Suite 19

FA_WebInterface 19

First Steps

General Information 19

Installation and Startup 20

Configuration 34

Different FF-versions 36

FlexFrame solution 20

generic services 42

Installation 22

EzSetup

Backup with EzBackup 25

General 24

Important 29

Standard Installation 26

Usage 32

EzSetup-Installation 24

Installation Packages 22

Requirements 22

rpm-Installation 23

start / stop 35

FlexFrame

accounting option 85

accounting Plug-in 197

autonomy 97

performance 85, 197

reporting Plug-in 197

Fujitsu FlexFrame scripts 127

Index


G

generic services 83

group schema 73

grouping 70

grouping function 73

H

hardware resources 70

I

introduction 65

IP storage 20

L

live cache 81

M

maintenance scripts 101

Messenger

Database connection 50

Messenger 50

Messenger

Database connection

mySQL 50

Messenger

Database connection

PostgreSQL 50

migration 150

migration tool 154

Multi Node Failure 102

LongTime 103

ShortTime 103

myAMC 14

myAMC.FA 14

myAMC.FA version management 52

Lifecycle management files 53

Version manager 56

Version-Components 55

Version-Dependency 54

Lifecycle process 52

myAMC.FA_AppAgent 23

myAMC.FA_CtrlAgent 23

myAMC.FA_LogAgent 14

myAMC.FA_Messenger 14, 19

N

Netapp Filer 20

node failure 102

NoWatch/Watch of SAP services 133

O

operating mode 125

event mode 125

reaction mode

central 126

local 125

P

parameterization

class creation rules 248

default parameter file 222, 228, 274

FA Agents 219

FlexFrame autonomy 249

grouping 223

pool creation 223

service classes 247

Index


service power value 247

service priority 247

traps 220

pool creation 70

pool-indepenednt-spare 116

Productive Pool 116

SparePool 117

power shutdown 199

Application Nodes 211

architecture 200

Blade systems 201

configuration 203

default shutdown mode 216

Management Blades 209

other Server 202

PRIMERGY systems 201

switchover control parameters 204

user, password, community 206

virtual VM Server 202

Powershutdownkonzept

für virtuelle Server 202

R

read/write Root Image 20

read-only Root Image 20

reboot 70

related documents 18

replicated enqueue service (ERS) 80

restart 70

S

SAP HANA 82

SAP start scripts 20

self-repair strategies 101

Server LAN 20

service classes 77

service detection model 84

service failure 101

service instance 77

service power value 77

service priotiy 77

service reaction model 84

service state model 83

SLES 20

Solution Manager Diagnostic

(SMD) 80

spare nodes 102

start/stop of SAP services 131

start/stop/status of Agents 129

Storage LAN 20

switchover 70

T

TakeOver Rules 109

Add Rule 110

Dynamic Rule 112

Dynamic Rule Example 115

Replace Rule 110

SpareNode 109

Substitution Rule 111

TakeOver Startegy 108

Index


testament types 78

U

Usage of Pool-Master 59

Automation of steps during pool

creation 62

Create Pool-Master from existing

pool 60

Create/Update pool from Pool-

Master 61

Usage of ‘fa_MasterPoolMgr.sh’ 62

use cases 134

Autonomy for Application

Instances 136

File System Utilization 138

High Availability and Autonomy

Rules 139

Performance, Capacity and

Accounting 138

semi-autonomous operation 135

W

WebInterface

BlackBoard settings 162

configuration 159

configuration of FlexFrame

autonomy 183

Hypervisor Monitor 163

info and help 196

interaction 194

login 164

message display 178

overview 165

paths and file names 163

pool / group tree 166, 193

reading out the FA data 163

starting 164

status display 172

updates 194

visualization 164