powerha systemmirror for linux version 7 · powerha systemmirror gui installation process from the...

IBM PowerHA SystemMirror for Linux Version 7.2

IBM

Note

Before using this information and the product it supports, read the information in “Notices” on page97.

This edition applies to IBM® PowerHA® SystemMirror® 7.2 for Linux and to all subsequent releases and modifications untilotherwise indicated in new editions.© Copyright International Business Machines Corporation 2017, 2019.US Government Users Restricted Rights – Use, duplication or disclosure restricted by GSA ADP Schedule Contract withIBM Corp.

Contents

About this document..............................................................................................vHighlighting...................................................................................................................................................vCase-sensitivity in Linux...............................................................................................................................vISO 9000.......................................................................................................................................................v

What's new........................................................................................................... 7

Concepts............................................................................................................. 11Compare PowerHA SystemMirror for AIX to PowerHA SystemMirror for Linux...................................... 11High availability clustering for Linux......................................................................................................... 13

High availability and hardware availability.......................................................................................... 13Benefits of PowerHA SystemMirror..................................................................................................... 13High availability clusters...................................................................................................................... 14

Physical components of a PowerHA SystemMirror cluster...................................................................... 14Nodes....................................................................................................................................................15Networks.............................................................................................................................................. 15Clients...................................................................................................................................................15

Managing users and user groups...............................................................................................................15Cluster nodes, networks, and heartbeating..............................................................................................18

Nodes....................................................................................................................................................18Cluster networks.................................................................................................................................. 18Configuring the netmon.cf file .............................................................................................................19IP address takeover............................................................................................................................. 20IP address takeover by using IP aliases..............................................................................................21Heartbeating over TCP/IP and disk......................................................................................................21Split policy............................................................................................................................................ 22Resources and resource groups.......................................................................................................... 25

Cluster configurations................................................................................................................................29Standby configurations........................................................................................................................ 29Takeover configurations.......................................................................................................................30

Planning..............................................................................................................35

Installing............................................................................................................ 39Planning the installation............................................................................................................................ 39Installing PowerHA SystemMirror ............................................................................................................ 40Cluster snapshot........................................................................................................................................ 41Upgrading PowerHA SystemMirror for Linux ........................................................................................... 41

Performing an upgrade operation on an offline cluster...................................................................... 42Performing a rolling upgrade operation ..............................................................................................42

Configuring......................................................................................................... 45Creating a cluster.......................................................................................................................................45Add a cluster.............................................................................................................................................. 46Configuring resources................................................................................................................................48Configuring resource group....................................................................................................................... 50Configuring a Shared File System resource ............................................................................................. 52Verifying the standard configuration......................................................................................................... 53Configuring dependencies between resource groups.............................................................................. 53

iii

Troubleshooting PowerHA SystemMirror.............................................................. 59Troubleshooting PowerHA SystemMirror clusters................................................................................... 59Using PowerHA SystemMirror cluster log files......................................................................................... 59Using the Linux log collection utility..........................................................................................................60Solving common problems........................................................................................................................ 60

PowerHA SystemMirror startup issues................................................................................................60PowerHA SystemMirror disk issues.....................................................................................................62PowerHA SystemMirror resource and resource group issues............................................................ 63PowerHA SystemMirror Fallover issues...............................................................................................64PowerHA SystemMirror additional issues........................................................................................... 65

PowerHA SystemMirror GUI.................................................................................67Planning......................................................................................................................................................67Installing.................................................................................................................................................... 69Logging in to the PowerHA SystemMirror GUI ......................................................................................... 70Navigating.................................................................................................................................................. 71Importing multiple clusters.......................................................................................................................73Smart Assist for SAP HANA....................................................................................................................... 74Troubleshooting.........................................................................................................................................75Configuring................................................................................................................................................. 77

Adding users and assigning roles........................................................................................................ 77Logging in as a non-root user...............................................................................................................78Configuring the PowerHA SystemMirror GUI server to be highly available........................................79Limiting access to the PowerHA SystemMirror GUI............................................................................80Restricting the use of IP address in the GUI....................................................................................... 81

Smart Assist for PowerHA SystemMirror Smart Assist for PowerHASystemMirror...................................................................................................83PowerHA SystemMirror for SAP HANA..................................................................................................... 83

Planning for SAP HANA........................................................................................................................ 84Configuring the Smart Assist application for SAP HANA.....................................................................85

PowerHA SystemMirror for SAP NetWeaver............................................................................................. 87Planning for SAP NetWeaver................................................................................................................87Installing an SAP system..................................................................................................................... 88Configuring SAP NetWeaver.................................................................................................................92

Troubleshooting PowerHA SystemMirror Smart Assist issues.................................................................95PowerHA SystemMirror is not able to harvest some values during Wizard execution.......................95Replication Site Name that is configured in Smart Assist wizard is different from Replication

Site Name shown in SAP HANA setup wizard................................................................................ 95Smart Assist policy fails to activate..................................................................................................... 95Smart Wizard does not detect or show one or more Ethernet interfaces in the list.......................... 95

Notices................................................................................................................97Privacy policy considerations.................................................................................................................... 98Trademarks................................................................................................................................................ 99

Index................................................................................................................ 101

iv

About this document

This document provides information about how you can configure, maintain, and monitor clusters withPowerHA SystemMirror for Linux.

HighlightingThe following highlighting conventions are used in this document:

Bold Identifies commands, subroutines, keywords, files, structures, directories, and otheritems whose names are predefined by the system. Bold highlighting also identifiesgraphical objects, such as buttons, labels, and icons that the you select.

Italics Identifies parameters for actual names or values that you supply.

MonospaceIdentifies examples of specific data values, examples of text similar to what youmight see displayed, examples of portions of program code similar to what you mightwrite as a programmer, messages from the system, or text that you must type.

Case-sensitivity in LinuxEverything in the Linux operating system is case-sensitive, which means that it distinguishes betweenuppercase and lowercase letters. For example, you can use the ls command to list files. If you type LS,the system responds that the command is not found. Likewise, FILEA, FiLea, and filea are threedistinct file names, even if they reside in the same directory. To avoid causing undesirable actions to beperformed, always ensure that you use the correct case.

ISO 9000ISO 9000 registered quality systems were used in the development and manufacturing of this product.

© Copyright IBM Corp. 2017, 2019 v

vi PowerHA SystemMirror for Linux Version 7.2

What's new in PowerHA SystemMirror 7.2 for Linux

Read about new or significantly changed information in PowerHA SystemMirror 7.2 for Linux.

How to see what's new or changed

To help you see where technical changes have been made, the information center uses:

• The image to mark where new or changed information begins.

• The image to mark where new or changed information ends.

What's new in PowerHA SystemMirror Version 7.2.3 for Linux with Service Pack 1PowerHA SystemMirror support on RHEL 7.7 and RHEL 8.0

PowerHA SystemMirror Version 7.2.3 for Linux with Service Pack 1 is supported on Red Hat EnterpriseLinux (RHEL) 7.7 and RHEL 8.0

PowerHA SystemMirror support on SLES 15 Service Pack 1PowerHA SystemMirror Version 7.2.3 for Linux with Service Pack 1 is supported on SUSE LinuxEnterprise Server (SLES) 15 Service Pack 1.

Support for Cost Optimized configuration of SAP HANAThe Cost Optimized configuration mode uses the secondary node for running a non-productive SAPHANA instance. The maximum amount of memory that is used by the productive SAP HANA instanceon the secondary node is limited by configuring a value for the Global Allocation Limitparameter and by disabling the table preload. Whenever takeover must occur, PowerHA SystemMirrorstops the non-productive SAP HANA instance first and then takeover occurs. For more information,see PowerHA SystemMirror for SAP HANA.

Multinode disk heartbeatPowerHA SystemMirror Version 7.2.3 for Linux with Service Pack 1, or later supports up to four nodesfor disk heartbeat. For more information, see Heartbeating over disk.

Enable cluster services to start or stop automatically on a node after system restartYou can bring the cluster services online or offline on a node after the system restarts by configuringthe START_ON_BOOT parameter. For more infraction see, Adding a node to a cluster.

Enhancements in Smart Assist

• If you specify the value for the DEL_POLICY parameter as YES in the Delete Smart Assist operation,the configured srHook files and configuration that is related to PowerHA SystemMirror are deleted.For more information, see SAP HANA operations.

• Enhanced the Setup Smart Assist operation for a Smart Assist policy by adding thePREFER_TAKEOVER parameter. The PREFER_TAKEOVER parameter defines whether PowerHASystemMirror must switch over the SAP HANA database to the secondary node when the primarySAP HANA instance fails instead of restarting the SAP HANA instance locally on the primary node.For more information, see PowerHA SystemMirror for SAP HANA.

Enhancements in PowerHA SystemMirror graphical user interface (GUI)The following updates are new in PowerHA SystemMirror GUI:

• Support for Cost Optimized configuration of SAP HANA.• Create and manage resource group dependencies.• Non-root support for creating clusters and cloning clusters from snapshots.• Configure PowerHA SystemMirror GUI server to be highly available by using the High Availability

wizard option in the PowerHA SystemMirror GUI.• Import multiple clusters by using the Add Multiple Clusters wizard.

© Copyright IBM Corp. 2017, 2019 7

• Enhanced activity log to display the activity ID, start time and end time of the activity, and theduration of the activity. You can also view details such as the number of successful login attemptsand failed login attempts, the number of new activities, and the number activities that are notviewed.

• Enhanced security features with options to disable anonymous login and global access.• Automatic download and installation of the remaining files that are required to complete the

PowerHA SystemMirror GUI installation process from the IBM website by using the smuiinst.kshcommand.

For more information, see PowerHA SystemMirror graphical user interface (GUI).

June 2019

The following information is a summary of the updates that are made to the PowerHA SystemMirror forLinux documentation:

• Smart Assist for SAP NetWeaver updates:

– Added information about Standalone Enqueue Server 2 support and updated the prerequisites forconfiguring SAP NetWeaver in the Planning for SAP NetWeaver topic.

– Added information about the INSTALL_TYPE and MODE parameters in the Configuring SAPNetWeaver topic.

– Updated information about the Delete Smart Assist operation in the SAP NetWeaver operations topic.– Added information about High availability of SAP NetWeaver on up to four-nodes in the Planning for

SAP NetWeaver topic.• Smart Assist for SAP HANA updates:

– Added information about SAP HANA Active/Active (read enabled) secondary configuration modesupport through Smart Assist in the PowerHA SystemMirror for SAP HANA topic.

– Updated information about the Delete Smart Assist operation in the SAP HANA operations topic.– Added information about the MODE and CONFIG parameters in the Configuring the Smart Assist

application for SAP HANA topic.• Added the Smart Assist for SAP HANA topic that describes the Smart Assist wizard that is available in

the PowerHA SystemMirror graphical user interface.• Added information about the Smart Assist wizard in the Navigating the PowerHA SystemMirror GUI

topic.• Updated information about the support for the following operating systems in the Planning for PowerHA

SystemMirror for Linux topic:

– SUSE Linux Enterprise Server (SLES) for SAP 12 SP4– SUSE Linux Enterprise Server (SLES) for SAP 15– Red Hat Enterprise Linux (RHEL) 7.6

• Updated migration information in the following topics:

– Upgrading PowerHA SystemMirror for Linux– Performing an upgrade operation on an offline cluster– Performing a rolling upgrade operation

• Updated the syntax for the file_collection, snapshot, and smartassist classes in the clmgrcommand topic.

June 2018

The following information is a summary of the updates that are made to the PowerHA SystemMirror forLinux documentation:

8 PowerHA SystemMirror for Linux Version 7.2

• Updated the following topics with information that is related to StartAfter relationship between theSAP NetWeaver system (application server) and the SAP HANA High Availability resources:

– Verifying the SAP NetWeaver High Availability• Added the Managing users and user groups topic that describes users and user groups management.• Added the /var/pha/log/appmon.log log file in the Using cluster log files topic.• Updated the following topics for the Shared Storage functionality:

– Highly available resources are in the Failed offline state– Resource group has Stuck Online state– Planning the installation of PowerHA SystemMirror for Linux– Installing PowerHA SystemMirror for Linux– PowerHA SystemMirror failed to create a cluster– Configuring a Shared File System resource– Verifying the standard configuration– Configuring resource groups for PowerHA SystemMirror for Linux– Configuring resources for PowerHA SystemMirror for Linux– Creating a cluster for PowerHA SystemMirror for Linux– File system

• Update the clmgr command with two new operations.• Updated information about resource groups in the clRGinfo command topic.

What's new in PowerHA SystemMirror 7.2 for Linux 9

PowerHA SystemMirror for Linux conceptsThe following information introduces important concepts you must understand before you can use thePowerHA SystemMirror for Linux.

Compare PowerHA SystemMirror for AIX to PowerHA SystemMirror for LinuxThe following table compares different functions that are shared between PowerHA SystemMirror forAIX® and PowerHA SystemMirror for Linux.

Table 1. Compare PowerHA SystemMirror for AIX to PowerHA SystemMirror for Linux

Functions PowerHA SystemMirror for AIX PowerHA SystemMirror for Linux

Installation The SP release contains only themodified filesets, so the base buildneeds to be installed beforeinstalling a service pack.

The PowerHA Linux release packaging forthe SP build is similar to the base build. So,there is no difference in the installation ofthe base and the SP build and for the SPbuild installation, you do not need to installthe base build first.

Configuration Any configuration change that isdone by the user is saved only inthe local Configuration Database(ODM) and is applied to thedifferent cluster nodes aftersynchronization procedure.

If any configuration changes are done bythe user, no separate synchronizationprocedure is needed and the configurationchange is applied immediately to all clusternodes.

Split policy(manual)

For the 2-node cluster in PowerHASystemMirror for AIX, if one of thenodes goes down, the other nodetakes over the resources of theresource group even when splitpolicy is set to the Manual state.

In PowerHA SystemMirror for Linux, whensplit policy is set to the Manual state for the2-node cluster, and if one node goes down,then manual intervention is needed byusing the runact command so that theresources of the resource group areacquired by the other node.

Split policy(manual)

When the split operation happens,the PowerHA SystemMirror for AIXsoftware broadcasts a message toall terminals that indicates splitoperation and starts the relevantscript to continue or to getrebooted as a recovery action.

When the split operation happens, the userneeds to verify the split operation by usingthe following command lssrc -lsIBM.RecoveryRM |grep"Operational Quorum State". Thevalue PENDING_QUORUM indicates splitoperation and the user needs to run therunact command with the relevant valueto continue or to get rebooted as a recoveryaction.

Split handling(tiebreaker)

The Tiebreaker policy can be usedwith any number of cluster nodes.

PowerHA SystemMirror for Linux usestiebreaker only when there is a tie and thetotal number of nodes that are configuredin a cluster is even.


Table 1. Compare PowerHA SystemMirror for AIX to PowerHA SystemMirror for Linux (continued)

Functions PowerHA SystemMirror for AIX PowerHA SystemMirror for Linux

Split handling(tiebreaker)

When a tiebreaker is configuredand more than half of the nodes arein the shutdown state (for example,two nodes are down in a 3-nodecluster), then also PowerHASystemMirror for AIX can start theresources on a node that is up andrunning.

PowerHA SystemMirror for Linux uses theNormal quorum type of Reliable ScalableCluster Technology (RSCT) to create a peerdomain cluster. This operation requires thatat least half-of-the nodes must be up. So iftwo nodes are down in a 3-node cluster,then even if a tiebreaker is configured, thenode that is up would not be able to acquirethe resources.

Stop behavior ofStartAfterdependency

The StartAfter dependency doesnot impact the stop behavior of theresource groups.

If two resource groups have StartAfterdependency, the target resource groupcannot be stopped while the sourceresource group is online.

Dependency (anti-collocated)

The resource groups with anti-collocated dependency are neveronline on the same node.

If the target resource group with anti-collocated dependency is brought onlinewhen the source resource group is alreadyonline on a node, and the intended node ofthe target resource group is not available,the resource group becomes online on thesame node where source resource group isalso active.

Dependency(collocated)

For two resource groups withcollocated dependency, failure ofany one of the resource groupstriggers movement of both theresource groups to another node.

For two resource groups with collocateddependency, if any one of the resourcegroups fails, then the PowerHASystemMirror software does not move boththe resource groups to another node.

unmanageparameter (clusteror node-levelparameter)

The unmanage parameter isavailable in the PowerHASystemMirror for AIX. Thisparameter allows the workload torun without being managed byPowerHA SystemMirror.

This parameter is not available in thePowerHA SystemMirror for Linux Version7.2.

START_ON_BOOTparameter (node-level parameter)

Using the START_ON_BOOTparameter, you can automaticallybring the cluster services online oroffline on a node after the nodereboot.

Using the START_ON_BOOT parameter, youcan automatically bring the cluster servicesonline or offline on a node after the nodereboot. The START_ON_BOOT parameter isavailable in PowerHA SystemMirror Version7.2.3 for Linux with Service Pack 1 or later.

Applicationconfiguration

The application controller and theapplication monitor are separatelycreated by using the clmgrfunction.

A single application class in clmgr is usedto configure the start, stop, and monitorscripts for a specific application.


High availability clustering for LinuxThe IBM PowerHA SystemMirror software provides a low-cost commercial computing environment thatensures quick recovery of mission-critical applications from hardware and software failures.

PowerHA SystemMirror monitors the cluster resources for failures, and when a problem is detected,PowerHA SystemMirror moves the application (along with resources that ensure access to theapplication) to another node in the cluster.

High availability and hardware availabilityHigh availability software is sometimes confused with simple hardware availability. Fault tolerant,redundant systems (such as RAID) and dynamic switching technologies (such as DLPAR) provide recoveryof certain hardware failures, but do not provide the full scope of error detection and recovery required tokeep a complex application highly available.

A modern, complex application requires access to all of these components:

• Nodes (CPU, memory)• Network interfaces (including external devices in the network topology)• Disk or storage devices• Application software

Surveys of the causes of downtime show that actual hardware failures account for only a smallpercentage of unplanned outages. Other contributing factors include:

• Operator errors• Environmental problems• Application and operating system errors.

Reliable and recoverable hardware simply cannot protect against failures of all these different aspects ofthe configuration. Keeping these varied elements, and therefore the application, highly available requires:

• Thorough and complete planning of the physical and logical procedures for access and operation of theresources on which the application depends. These procedures help to avoid failures in the first place.

• A monitoring and recovery package that automates the detection and recovery from errors.• A well-controlled process for maintaining the hardware and software aspects of the clusterconfiguration while keeping the application available.

Benefits of PowerHA SystemMirrorPowerHA SystemMirror has many benefits.

PowerHA SystemMirror helps you with the following:

• The PowerHA SystemMirror planning process and documentation include tips and advice on the bestpractices for installing and maintaining a highly available PowerHA SystemMirror cluster.

• Once the cluster is operational, PowerHA SystemMirror provides the automated monitoring andrecovery for all the resources on which the application depends.

• PowerHA SystemMirror provides a full set of tools for maintaining the cluster while keeping theapplication available to clients.

PowerHA SystemMirror allows you to:

• Quickly and easily setup a cluster by using the clmgr command-line interface or the applicationconfiguration assistants (Smart Assists).

• Ensure high availability of applications by eliminating single points of failure in a PowerHA SystemMirrorenvironment.

• Manage how a cluster handles component failures.

PowerHA SystemMirror for Linux concepts 13

• Secure cluster communications.• Monitor PowerHA SystemMirror components and diagnose problems that might occur.

High availability clustersAn application cluster is a group of loosely coupled machines networked together, sharing disk networkand other resources.

In a high availability cluster, multiple server machines cooperate to provide a set of services or resourcesto clients.

Physical components of a PowerHA SystemMirror clusterPowerHA SystemMirror provides a highly available environment by identifying a set of resources essentialto uninterrupted processing of an application. It also defines a protocol that nodes use to collaborate toensure that these resources are available.

PowerHA SystemMirror extends the clustering model by defining relationships among cooperatingprocessors where one processor provides the service offered by a peer should the peer be unable to doso. As shown in the following figure, a PowerHA SystemMirror cluster is made up of the following physicalcomponents:

• Nodes• Networks• Clients.

The PowerHA SystemMirror software allows you to combine physical components into a wide range ofcluster configurations, providing you with flexibility in building a cluster that meets your processing andavailability requirements. This figure shows one example of a PowerHA SystemMirror cluster. OtherPowerHA SystemMirror clusters could look very different - depending on the number of processors, thechoice of networking and disk technologies, and so on.

Figure 1. Example of a PowerHA SystemMirror cluster


PowerHA SystemMirror nodesNodes form the core of a PowerHA SystemMirror cluster. A node is a processor that runs Linux, thePowerHA SystemMirror software, and the application software.

In a PowerHA SystemMirror cluster, each node is identified by a unique name. A node has access to a setof resources: networks, network addresses, and applications. Typically, a node runs a server or a backend application that accesses data on the shared external disks.

NetworksAs an independent, layered component of the Linux operating system, the PowerHA SystemMirrorsoftware is designed to work with any TCP/IP-based network.

Nodes in a cluster use the network to:

• Allow clients to access the cluster nodes• Enable cluster nodes to exchange heartbeat messages

Types of networks

The PowerHA SystemMirror software defines two types of communication networks, characterized bywhether these networks use communication interfaces based on the TCP/IP subsystem (TCP/IP-based)or disk-based networks.

TCP/IP-based networkConnects two or more server nodes, and optionally allows client access to these cluster nodes, usingthe TCP/IP protocol.PowerHA SystemMirror uses only unicast communication for heartbeat.

Disk heartbeatProvides communication between PowerHA SystemMirror cluster nodes to monitor the health of thenodes, networks and network interfaces, and to prevent cluster partitioning

ClientsA client is a processor that can access the nodes in a cluster over a local area network.

Clients each run a "front end" or client application that queries the server application running on thecluster node. The PowerHA SystemMirror software provides a highly available environment for criticaldata and applications on the cluster nodes. The PowerHA SystemMirror software does not make theclients themselves highly available.

Managing users and user groupsThe clmgr command manages user accounts and user groups on all nodes in a cluster by makingconfiguration changes on any node in a cluster. The clmgr command also supports Lightweight DirectoryAccess Protocol (LDAP) for users and user groups management.

PowerHA SystemMirror manages Linux and LDAP users and user groups across all PowerHA SystemMirrorclusters. PowerHA SystemMirror user groups provide an additional level of security and enable thesystem administrators to change the configuration settings of a group of users as a single entity.

Managing Linux and LDAP user accounts in a PowerHA SystemMirror cluster

You can use the clmgr command to manage users and user groups from any node in a cluster. If youcreate a user that already exist on any node in the cluster, the operation might fail.

The following Linux files, which store user account information, must be consistent across all clusternodes.

• The /etc/passwd system file• Other system files in the /etc/security directory


Note: If a cluster node fails, users can log on to the operating nodes without experiencing any issues thatare caused by mismatched user or group IDs.

As the system administrator of a PowerHA SystemMirror, you can use the clmgr command to manageuser and user groups from any node in a cluster. The clmgr command propagates new and updatedinformation to all of the other nodes in the cluster.

To configure user accounts on an LDAP server, you must have an installation of the following software:

• openLDAP Server• openLDAP Client

To add a user on the local system, enter the following command:

clmgr add user Local_User

where,

• The Local_User parameter is the user name of the user account that must be added on the local systemand on the other nodes of the PowerHA SystemMirror cluster.

Note: If a node in a cluster is in an Offline state, you cannot create a user on that node.

To add an LDAP user on the local system, enter the following command:

clmgr add user LDAP_User registry=ldap DN=

where,

• The DN parameter is the distinguished name, which is created when you set up the openLDAP server.The clmgr command prompts for an LDAP password, which is the LDAP administrator password usedwhen you set up the openLDAP server.

The clmgr add user command includes the following attributes for the Local user or an LDAP user:

Table 2. Local user and LDAP user attributes

Field name Description

user_name A login name that identifies this user account onthe system.

ID Defines a unique decimal integer string toassociate with this user account on the system.

ADMINSTRATIVE Select True if this user will be an administrativeuser. Otherwise, select False.

CHANGE_ON_NEXT_LOGIN Forces user to change the password on next login.

PRIMARY Specifies the group name of the user to which theuser belongs when the user logs in for the firsttime.

GROUPS Specifies name of all user groups to which a userbelongs. A user can access authority to theprotected resources.

ROLES Creates and manages role-based access control(RBAC) roles for users and user groups. You canuse these roles to control which commands can beexecuted by different sets of users of PowerHASystemMirror.


Table 2. Local user and LDAP user attributes (continued)


REGISTRY Indicates where the information about the newuser must be stored. If the user is being definedlocally and within the Linux environment, the valueLOCAL must be specified. If the user is defined ona remote server, such as the LDAP server, the valueLDAP must be used.

DN A DN is a sequence of relative distinguished names(RDN) separated by commas. The DN attribute isrelevant when the registry attribute is set to LDAP.

HOME Specifies the home directory of a user.

SHELL Specifies the default shell for user.

EXPIRATION Specifies the expiration date of the password foruser.

LOCKED Locks the password of the specified account.

DAYS_TO_WARN Defines the number of days before the systemissues a warning that a password change isrequired.

LOCKOUT_DELAY Specifies the time period, in weeks, betweenpassword expiration and lockout.

MAX_PASSWORD_AGE Defines the maximum age (in weeks) for the userpassword.

MIN_PASSWORD_AGE Defines the minimum age (in weeks) for the userpassword.

Managing user group accounts in a PowerHA SystemMirror cluster

Similar to users, the user groups can be configured on PowerHA SystemMirror Linux setup on the localsystem (files) and on an LDAP server.

To configure user accounts on an LDAP server, you must have an installation of the following software:

• openLDAP Server• openLDAP Client

To add a group on the local system, enter the following command:

clmgr add group User_group

The clmgr` command adds a group on the local system and on the other nodes of the cluster.

Note: The user groups are not created on the nodes, which are in an Offline state in the cluster.

To add a group in LDAP, enter the following command:

clmgr add group User_group registry=ldap DN=

The following attributes can be used to add a group:


Table 3. Group attributes


ID Defines a unique decimal integer string toassociate with the user account on the system.

ADMINSTRATIVE Select True if this user will be an administrativeuser. Otherwise select False.

USERS Specifies the names of the users that belong touser group. The members of a group can access(that is, read, write, or run) a resource or file that isowned by another member of the group asspecified by the access control list of the resource.

REGISTRY Indicates where the information about the newuser must be stored. If the user is being definedlocally and within the Linux environment, the valueLOCAL must be specified. If the user is defined ona remote server, such as the LDAP server, the valueLDAP must be used.

DN A DN is a sequence of relative distinguished names(RDN) separated by commas. The DN attribute isrelevant when the registry attribute is set to LDAP.

PowerHA SystemMirror cluster nodes, networks, and heartbeating conceptsThis section introduces major cluster topology-related concepts and definitions that are used throughoutthe documentation and in the PowerHA SystemMirror user interface.

NodesA node is a processor that runs both Linux and the PowerHA SystemMirror software.

Nodes might share a set of resources such as, networks, network IP addresses, and applications. ThePowerHA SystemMirror software supports up to 4 nodes in a cluster. In a PowerHA SystemMirror cluster,each node is identified by a unique name. In PowerHA SystemMirror, a node name and a hostname mustbe the same. Nodes serve as core physical components of a PowerHA SystemMirror cluster.

Cluster networksCluster nodes communicate with each other over communication networks.

If one of the physical network interface cards on a node on a network fails, PowerHA SystemMirrorpreserves the communication to the node by transferring the traffic to another physical network interfacecard on the same node. If a "connection" to the node fails, PowerHA SystemMirror transfers resources toanother node to which it has access.

In addition, the clustering software sends heartbeats between the nodes over the cluster networks toperiodically check on the health of the cluster nodes themselves. If the clustering software detects noheartbeats from a node, a node is considered as failed and resources are automatically transferred toanother node.

It is highly recommend configuring multiple communication paths between the nodes in the cluster.Having multiple communication networks prevents cluster partitioning, in which the nodes within eachpartition form their own entity. In a partitioned cluster, it is possible that nodes in each partition couldallow simultaneous non-synchronized access to the same data. This can potentially lead to differentviews of data from different nodes.


Physical and logical networksA physical network connects two or more physical network interfaces.

Note: If you are considering a cluster where the physical networks use external networking devices toroute packets from one network to another, consider the following: When you configure a PowerHASystemMirror cluster, PowerHA SystemMirror verifies the connectivity and access to all interfaces definedon a particular physical network. However, PowerHA SystemMirror cannot determine the presence ofexternal network devices such as bridges and routers in the network path between cluster nodes. If thenetworks have external networking devices, ensure that you are using devices that are highly availableand redundant so that they do not create a single point of failure in the PowerHA SystemMirror cluster.

A logical network is a portion of a physical network that connects two or more logical network interfacesor devices. A logical network interface or device is the software entity that is known by an operatingsystem. There is a one-to-one mapping between a physical network interface/device and a logicalnetwork interface/device. Each logical network interface can exchange packets with each logical networkinterface on the same logical network.

If a subset of logical network interfaces on the logical network needs to communicate with each other(but with no one else) while sharing the same physical network, subnets are used. A subnet mask definesthe part of the IP address that determines whether one logical network interface can send packets toanother logical network interface on the same logical network.

Logical networks in PowerHA SystemMirrorPowerHA SystemMirror has its own, similar concept of a logical network.

All logical network interfaces in a PowerHA SystemMirror network can communicate PowerHASystemMirror packets with each other directly. Each logical network is identified by a unique name. APowerHA SystemMirror logical network might contain one or more subnets.

PowerHA SystemMirror communication interfacesA PowerHA SystemMirror communication interface is a grouping of a logical network interface, a serviceIP address and a service IP label that you defined in PowerHA SystemMirror.

PowerHA SystemMirror communication interfaces combine to create IP-based networks. A PowerHASystemMirror communication interface is a combination of:

• A logical network interface is the name to which Linux resolves a port (for example, eth0) of a physicalnetwork interface card.

• A service IP address is an IP address (for example, 129.9.201.1) over which services, such as anapplication, are provided, and over which client nodes communicate.

• A service IP label is a label (for example, a hostname in the /etc/hosts file, or a logical equivalent of aservice IP address, such as node_A_en_service) that maps to the service IP address.

Communication interfaces in PowerHA SystemMirror are used in the following ways:

• A communication interface refers to IP-based networks and network interface cards (NIC). The NICsthat are connected to a common physical network are combined into logical networks that are used byPowerHA SystemMirror.

• Each NIC is capable of hosting several TCP/IP addresses. When configuring a cluster, you must definethe IP addresses that PowerHA SystemMirror monitors (base or boot IP addresses), and the IPaddresses that PowerHA SystemMirror keeps highly available (the service IP addresses).

• Heartbeating in PowerHA SystemMirror occurs over communication interfaces. PowerHA SystemMirroruses the heartbeating facility of RSCT. For more information, see the “Heartbeating over TCP/IP anddisk” on page 21 topic.

Configuring the netmon.cf fileThe netmon.cf file is an optional configuration file that customers can use to augment the pingoperation on the available target hosts on the network. The target hosts that are not defined to be part of


the cluster are not available from the cluster nodes. The target hosts can be accessed from the IPaddresses being monitored by topology services.

If you are running a single-node or two-node cluster, you must configure netmon.cf file to detect thenetwork interface failures.

PowerHA SystemMirror periodically attempts to contact each network interface in the cluster. If theattempt to contact an interface fails on one node of a two-node cluster, the corresponding interface onthe other node is also flagged as an Offline node. The other node is flagged as an Offline node because itdoes not receive a response from the peer cluster node. To avoid such behavior, the PowerHASystemMirror must be configured to contact a network instance outside of the cluster. You can use thedefault gateway of the subnet from which the PowerHA SystemMirror GUI is used.

To configure the netmon.cf file, complete the following steps:

1. Configure the netmon.cf file to check the status of the network monitored by the virtual switch.2. On each node, create the following file: /var/ct/cfg/netmon.cf

Note: Each line of netmon.cf file contains the system name or the IP address of the external networkinstance. IP addresses can be specified in the dotted decimal notation.

Example of the netmon.cf file

The following example shows the configuration result for the netmon.cf file.

#This is default gateway for all interfaces in the subnet 192.168.1.0 192.168.1.1

If you are using the Virtual I/O Server (VIOS), the configuration test becomes unreliable because thenetmon.cf file cannot determine whether the inbound traffic is received from the VIOS or a client. TheLPAR cannot distinguish a virtual adapter and a real adapter. To address this problem, the netmon librarysupports up to 32 targets for each local network adapter. If the ping operation for any of these targets issuccessful, the local adapter is considered to be in an Online state. The targets can be specified in thenetmon.cf file with the !REQD keyword. For example:

!REQD

The targets can be also specified in the netmon.cf file by entering the following command:

!IBQPORTONLY !ALL

Location/var/ct/cfg/netmon.cf

Location of the netmon.cf file in a PowerHA SystemMirror environment.

IP address takeoverIP address takeover is a mechanism for recovering a service IP label by moving it to another networkinterface card (NIC) on another node, when the initial NIC fails.

IPAT occurs if the physical network interface card on one node fails and if there are no other accessiblephysical network interface cards on the same network on the same node. Therefore, swapping IP labelsof these NICs within the same node cannot be performed and PowerHA SystemMirror will use IPAT torecover the service IP address by using a NIC on a backup node. IP address takeover keeps the IPaddress highly available by recovering the IP address after failures. PowerHA SystemMirror uses amethod called IPAT via IP aliases.


IP address takeover by using IP aliasesDefining IP aliases to network interfaces allows creation of more than one IP label and address on thesame network interface.

When a resource group containing the service IP label falls over from the primary node to the target node,the service IP labels are added (and removed) as alias addresses on top of the base IP addresses on anavailable NIC. This allows a single NIC to support more than one service IP label placed on it as an alias.Therefore, the same node can host more than one resource group at the same time.

When there a multiple interfaces on the same node connected to the same network, and those interfacesare not combined into a Ethernet Aggregation, all boot addresses must all be on different subnets. Also,any persistent addresses or service addresses must be on different subnets than the boot addresses.

Because IP aliasing allows coexistence of multiple service labels on the same network interface, you canuse fewer physical network interface cards in your cluster. Upon fallover, PowerHA SystemMirror equallydistributes aliases between available network interface cards.

Heartbeating over TCP/IP and diskA heartbeat is a type of a communication packet that is sent between nodes. Heartbeats are used tomonitor the health of the nodes, networks and network interfaces, and to prevent cluster partitioning.

In order for a PowerHA SystemMirror cluster to recognize and respond to failures, it must continuallycheck the health of the cluster. Some of these checks are provided by the heartbeat function.

Each cluster node sends heartbeat messages at specific intervals to other cluster nodes, and expects toreceive heartbeat messages from the nodes at specific intervals. If messages stop being received,PowerHA SystemMirror recognizes that a failure has occurred. Heartbeats can be sent over:

• TCP/IP networks• A physical volume (disk) which is accessible from all clusters nodes

The heartbeat function is configured to use specific paths between nodes. This allows heartbeats tomonitor the health of all PowerHA SystemMirror networks and network interfaces, as well as the clusternodes themselves.

The heartbeat paths are set up automatically by RSCT; you have the option to configure disk paths as partof PowerHA SystemMirror configuration.

Heartbeating over Internet Protocol networksPowerHA SystemMirror relies on RSCT to provide heartbeating between cluster nodes over InternetProtocol networks.

Heartbeating between cluster nodes can be configured by using the following command:

clmgr add network netw_0 TYPE=ether pvid=qoPDC7-G169-UJv3-zHI5-KePI-Whjy-K73nrb nodes= nodeA, nodeB

Heartbeating over diskYou can configure a disk heartbeat optionally. After configuration, the Reliable Scalable ClusterTechnology (RSCT) automatically exchanges heartbeat over the shared disk. These connections canprovide an alternative heartbeat path for a cluster that uses a single TCP/IP-based network.

You can configure a disk heartbeat by using following command:

clmgr add network netw_0 TYPE=disk pvid=qoPDC7-G169-UJv3-zHI5-KePI-Whjy-K73nrb nodes= nodeA, nodeB

Once the disk heartbeat configuration is complete, you can verify by using following command:

clmgr query network netw_0

To check whether the heartbeat interface is properly functioning, run the following command:


lssrc -ls cthats

The following output is generated:

Subsystem Group PID Statuscthats cthats 372762 active

Network Name Indx Defd Mbrs St Adapter ID Group IDdhbTEST_CG [1] 2 2 S 255.255.10.1 255.255.10.1dhbTEST_CG [1] dhb-1-2 0x01c8f4fe 0x01c8f511

In the specified sample command output, find the disk heartbeat stanza by locating the name of the diskheartbeat network, which is obtained by using the clmgr query network command output. The nameof the disk heartbeat interface resource is also displayed.

Positive result: Disk heartbeat is occurring if the Defd and Mbrs parameters are the same number:

Defd Mbrs2 2

Negative or error result: Disk heartbeat is not occurring if the Defd and Mbrs parameters are different:

Defd Mbrs2 1

If you get negative or error result for the disk heartbeat process, recheck the following options:

• Port VLAN ID (PVID) on which disk heartbeat network is configured is available on both the clusternodes.

• Storage connectivity with both the cluster nodes is working.• Physical volume on which disk heartbeat is configured is not being used by any other process, which

might block the IO on the device.

To remove the existing heartbeat network, enter the following command:

clmgr delete network netw_0

To modify an existing heartbeat network, enter the following command:

clmgr modify network netw_0 node=nodeA, nodeB, nodeC

Where, the node parameter lists the node that must be modified. PowerHA SystemMirror supports diskheartbeat for a four-node cluster.

You can use the clmgr modify network command only for the disk type of network.

Split policyA cluster split event can occur when a group of nodes cannot communicate with the remaining nodes in acluster. A cluster split event splits the cluster into two or more partitions.

After a cluster split event, the subdomain that has most of nodes are retained and the other subdomainsare deleted. If exactly half of the nodes of a domain are online and the remaining nodes are inaccessible,PowerHA SystemMirror must determine which subdomain has operational quorum by using ReliableScalable Cluster Technology (RSCT). The subdomain with an operational quorum is retained and the othersubdomains are deleted.

You can use PowerHA SystemMirror to configure a split policy that specifies the response to a cluster splitevent.

To configure a split policy, the following options are available:


None

The None split policy allows each partition that is created by the cluster split event to become anindependent cluster and each partition is started independently of the other partition. A user cancheck the quorum status by using the following command: lssrc -ls IBM.RecoveryRM

Before a split event, operational quorum state is in HAS_QUORUM state and the configuration quorumis TRUE. A user can do both operational change such as moving the cluster online or offline andconfiguration change such as adding resource group or deleting resource group into the cluster.

However, after the spilt event, for each partition, the operational quorum state will be HAS_QUORUMbut configurational quorum state will be FALSE. Hence, PowerHA SystemMirror and the user canperform operational change but not the configurational change.

For example, in two-nodes cluster, if the cluster split event occurs, all the resources are online onboth the nodes. As both the nodes have quorum, if the merge event occurs, lower priority node isrebooted if a critical resource is running on it and only RSCT subsystems is restarted if a non-criticalresource or no resource is running on it. You must use the None split policy if a user prefers multipleinstances of an application that runs simultaneously after split event.

To set the cluster policy as None, enter the following command:

clmgr modify cluster SPLIT_POLICY=none

Manual

The Manual split policy allows each partition that is created by the cluster split event to become anindependent cluster. However, each partition cannot start a workload, until the user allots theownership to the subcluster. Till then the Operational quorum state is in PENDING_QUORUM for eachsubcluster. Hence, PowerHA SystemMirror does not perform any action. The user can provideownership by using the following command:

runact -c IBM.PeerDomain ResolveOpQuorumTie Ownership=0/1

where1

Changes the quorum state from PENDING_QUORUM to HAS_QUORUM0

Changes the quorum state from PENDING_QUORUM to NO_QUORUM

The subcluster with NO_QUORUM state will have no authority to do any operational change on thenodes. If critical resource is running on NO_QUORUM node, the node is rebooted to avoid corruption ofcritical resource. The subcluster with HAS_QUORUM state has authority to do any operational changeon the subcluster.

Before a split event, the operational quorum state is HAS_QUORUM and configuration quorum is TRUE.PowerHA SystemMirror allows both operational change and configurational change into the cluster.

But after the split event, for each operation of the partition, the quorum state will bePENDING_QUORUM and configurational quorum will be FALSE. PowerHA SystemMirror does not allowthe user to do any operational change not even the configurational changes.

If a cluster splits, all the resources are in same state as they were before the split event. The user cangive permission as 1 to one subcluster and 0 to another cluster, so that the resources run only on onesub cluster. After giving permission, the PENDING_QUORUM will change to either HAS_QUORUM orNO_QUORUM.

After the split event, the node that has critical resource running on it is rebooted if 0 permission isgiven. If the non-critical resource or no resource is running on that subcluster, it remains same.

Manual Policy can be used if a user to give permission after split event before creation of anotherinstance of application.


The subcluster with HAS_QUORUM state automatically allows a user to perform any operationalchanges if the merge event occurs,. The lower priority node is rebooted if a critical resource is runningon it with an operational quorum state as HAS_QUORUM before merge and restarts the subsystemswhen the non-critical resource or no resource is running on it.

To set the cluster policy as Manual, enter the following command:

clmgr modify cluster SPLIT_POLICY=manual

Tiebreaker

You can use the tiebreaker option to specify a SCSI disk or a Network File System (NFS) file that isused by the split and merge policies.

A tiebreaker disk or an NFS file is used when the sites in the cluster can no longer communicate witheach other. This communication failure results in the cluster splitting the sites into two, independentpartitions. If failure occurs because the cluster communication links are not responding, bothpartitions attempt to lock the tiebreaker disk or the NFS file. The partition that acquires the tiebreakerdisk continues to function, while the other partition reboots, or has cluster services restarted,depending on if any critical resources are configured.

The disk or NFS-mounted file that is identified as the tiebreaker must be accessible to all nodes in thecluster.

Split Policy Tie Breaker allow one of the partitions that is created by the cluster split event to becomean independent cluster. After winning the tiebreaker, subcluster can start the workload or can allowthe user to perform an operational change. Whenever a split event occurs, the operational quorumstate remains in the PENDING_QUORUM state till one of the subclusters wins the tie breaker. Thesubcluster, which won the tie breaker starts the workload or allow the user to do operational changes.

The following are types of tiebreaker in PowerHA SystemMirror for Linux:NFS tiebreaker

When Network File System (NFS) type of tiebreaker is configured, one directory of the NFS servergets mounted on all nodes of the cluster. When a split event occurs due to network failure on onenode, that node loses the mounting of the NFS server directory and becomes the losing node. Theother node is the winning node.

Before you configure the NFS tiebreaker, a user must have proper understating like permission,accessibility of the NFS server machine and must check the directory on the NFS server that isgoing to be mounted on the local directory of the nodes. To configure the NFS tiebreaker inPowerHA SystemMirror for Linux, enter the following command:

clmgr modify cluster clMain SPLIT_POLICY=tiebreaker TIEBREAKER=nfs NFS_SERVER=192.2.2.5 NFS_SERVER_MOUNT_POINT=/test_nfs_tie NFS_LOCAL_MOUNT_POINT=/test_nfs_local NFS_FILE_NAME=nfsTestConfig

where,

• SPLIT_POLICY=tiebreaker is a type of split policy.• TIEBREAKER=nfs is type of tiebreaker.• NFS_SERVER=192.2.2.5 is the IP of nfs-server machine.• NFS_SERVER_MOUNT_POINT=/test_nfs_tie is the directory on the nfs-server machine.• NFS_LOCAL_MOUNT_POINT=/test_nfs_local is the directory on the nodes.• NFS_FILE_NAME=nfsTestConfig is any file name given by then user.

Before the split event, the quorum state can be seen by using the following command:

lssrc -ls IBM.RecoveryRMOperational Quorum State : HAS_QUORUMIn Config Quorum : TRUE


After the split event occurs, the quorum status will be:

Operational Quorum State : PENDING_QUORUMIn Config Quorum : FALSE

The quorum shows the following state after winning the tiebreaker on the winning node:

Operational Quorum State : HAS_QUORUMIn Config Quorum : FALSE

The quorum shows the following state after losing the tiebreaker on the losing node:

Operational Quorum State : NO_QUORUM In Config Quorum : FALSE

The losing node is rebooted if the critical resource is running on it. Otherwise, it remains as it is.

The winning node starts the workload if not running before otherwise it is running as it is and usercan perform operational configuration changes. The losing node will merge automatically to thecluster after reboot if a critical resource is running on it. If a non-critical resource is running on thenode, the RSCT subsystem is restarted on that node after the merge event.

The NFS tiebreaker must be used if a single-node interface fails. If two-nodes are connected totwo different switches and if one switch goes down, the node that is connected to that switch willnot be able to communicate. However, the second node can still communicate because the otherswitch is working. The second node wins as it can reach the NFS server.

Disk tiebreaker

Disk tiebreaker uses disk to resolve the tie. When the cluster split event occurs, each node tries tomake a lock on the disk. The node, which locks the disk first becomes the winning node. The othernode is the losing node.

Before you configure the disk tiebreaker, the user must find out about the shared disk among thenodes by using the following command:

lsrsrc IBM.Disk

To configure the disk tiebreaker in PowerHA SystemMirror for Linux, enter the followingcommand:

clmgr modify cluster SPLIT_POLICY=tiebreaker TIEBREAKER=disk DISK_WWID=36017595807eed37r0000000000000045

Where

SPLIT_POLICY=tiebreaker is type of split policy TIEBREAKER=disk is type of tiebreakerDISK_WWID=36017595807eed37r0000000000000045 is the shared disk used for tiebreaker.

The disk tiebreaker must be used if interface failure occurs on both the nodes. If both the nodesare connected to the same switch and if the switch goes down, the two nodes cannot reach theNFS serve r. The winning node is decided by the locking system on the disk. The node that locksthe disk first is the winning node.

PowerHA SystemMirror resources and resource groupsThis topic includes resource-related concepts and definitions that are used throughout thedocumentation and also in the PowerHA SystemMirror user interface.

Identifying and keeping your cluster resourcesThe PowerHA SystemMirror software provides a highly available environment.

The PowerHA SystemMirror software does this by:


• Identifying the set of cluster resources that are essential to the operation of an application, andcombining those resources into a resource group.

• Defining the resource group policies and attributes that dictate how PowerHA SystemMirror managesresources to keep them highly available during different cluster events like startup, fallover, andfallback.

By identifying resources and defining resource group policies, the PowerHA SystemMirror software makesnumerous cluster configurations possible, providing tremendous flexibility in defining a clusterenvironment tailored to individual requirements.

Identifying cluster resources

Cluster resources can include both hardware resources and software resources:

• Service IP labels or addresses• Applications

The PowerHA SystemMirror software handles the resource group as a unit, thus keeping theinterdependent resources together on one node and keeping them highly available.

Types of cluster resourcesThis section provides a brief overview of the resources that you can configure in PowerHA SystemMirrorand include into resource groups to let PowerHA SystemMirror keep them highly available.

ApplicationsThe purpose of a highly available system is to ensure that critical services are accessible to users.Applications usually need no modification to run in the PowerHA SystemMirror environment. Anyapplication that can be successfully restarted after an unexpected shutdown is a candidate for PowerHASystemMirror.

For example, all commercial DBMS products provide a checkpoint on the state of the disk in some sort oftransaction journal. In the event of a server failure, the fallover server restarts the DBMS, whichreestablishes database consistency and then resumes processing.

Note: The start and stop scripts are the main points of control for PowerHA SystemMirror over anapplication. It is very important that the scripts you specify operate correctly to start and stop all aspectsof the application. If the scripts fail to properly control the application, other parts of the applicationrecovery might be affected. For example, if the stop script you use fails to completely stop the applicationand a process continues to access a disk, PowerHA SystemMirror will not be able to recover it on thebackup node.

Add your application to a PowerHA SystemMirror resource group only after you have thoroughly testedyour application start and stop scripts.

The resource group that contains the application should also contain all the resources that the applicationdepends on, including service IP addresses. Once such a resource group is created, PowerHASystemMirror manages the entire resource group and, therefore, all the interdependent resources in it asa single entity. PowerHA SystemMirror coordinates the application recovery and manages the resourcesin the order that ensures activating all interdependent resources before other resources.

PowerHA SystemMirror includes application monitoring capability, whereby you can define a monitor todetect the unexpected termination of a process or to periodically poll the termination of an applicationand take automatic action upon detection of a problem.

Service IP labels and IP addressesA service IP label is used to establish communication between client nodes and the server node. Services,such as a database application, are provided using the connection made over the service IP label.

A service IP label can be placed in a resource group as a resource that allows PowerHA SystemMirror tomonitor its health and keep it highly available, either within a node or between the cluster nodes bytransferring it to another node in the event of a failure.

Note: Certain subnet requirements apply for configuring service IP labels.


Persistent node IP labelsA persistent node IP label is a useful administrative tool that lets you contact a node even if the PowerHASystemMirror cluster services are down on that node.

When you define persistent node IP labels PowerHA SystemMirror attempts to put an IP address on thenode. Assigning a persistent node IP label to a network on a node allows you to have a node-bound IPaddress on a cluster network that you can use for administrative purposes to access a specific node in thecluster. A persistent node IP label is an IP alias that can be assigned to a specific node on a clusternetwork and that:

• Always stays on the same node (is node-bound )• Co-exists on a network interface card that already has a service IP label defined• Does not require installing an additional physical network interface card on that node

There can be one persistent node IP label per node.

File systemA file system is a simple database for storing files and directories. A file system in Linux is stored on asingle logical volume.

The main components of the file system are the logical volume that holds the data and the file system log.PowerHA SystemMirror supports both ext3 and xfs as shared file systems.

Support of ext4 file system has been added in PowerHA SystemMirror version 7.2.2 for Linux with ServicePack 2 release.

Cluster resource groupsTo be made highly available by the PowerHA SystemMirror software, each resource must be included in aresource group. Resource groups allow you to combine related resources into a single logical entity foreasier management.

You can configure the cluster so that certain applications stay on the same node, or on different nodes notonly at startup, but during fallover and fallback events. To do this, you configure the selected resourcegroups as part of a location dependency set.

Participating node listA participating node list defines a list of nodes that can host a particular resource group.

You define a node list when you configure a resource group. The participating node list can contain someor all nodes in the cluster.

Typically, this list contains all nodes sharing the same data and disks.

Default node priorityDefault node priority is identified by the position of a node in the node list for a particular resource group.

The first node in the node list has the highest node priority. This node is also called the home node for aresource group. The node that is listed before another node has a higher node priority than the currentnode.

Home nodeThe home node (the highest priority node for this resource group) is the first node that is listed in theparticipating node list for a nonconcurrent resource group.

The home node is a node that normally owns the resource group.

The term home node is not used for concurrent resource groups because they are owned by multiplenodes.


Startup, fallover, and fallbackPowerHA SystemMirror ensures the availability of cluster resources by moving resource groups from onenode to another when the conditions in the cluster change.Cluster startup

When cluster services are started. resource groups are activated on different cluster nodes accordingthe resource group startup policy you selected.

Node failureResource groups that are active on this node fall over to another node.

Node recoveryWhen cluster services are started on the home node after a failure, the node reintegrates into thecluster, and may acquire resource groups from other nodes depending on the fallback policy for thegroup.

Resource failure and recoveryA resource group might fall over to another node, and be reacquired, when the resource becomesavailable.

Cluster shutdownWhen you stop cluster services you can choose to have the resource groups move to a backup node orbe taken offline on the current node.

Resource group policies

The following table describes the specific options for each policy:Startup

Startup is the activation of a resource group on a node (or multiple nodes). Resource group startupoccurs when cluster services are started.

Fallover

Fallover is the movement of a resource group from the node that currently owns the resource group toanother active node after the current node experiences a failure.

Fallover only occurs with nonconcurrent resource groups. concurrent resource groups are active on allnodes concurrently, so the failure of a single nodes means only that instance of the resource group onthat node is affected.

Fallback

Fallback is the movement of resources to the home node when it is reintegrated in the cluster afterfailure. No fallback means that resources continue to run on the same node, even after thereintegration of home node post failure.

Each combination of these policies allows you to specify varying degrees of control over which node,or nodes, control a resource group.

Startup, fallover, and fallback are specific behaviors that describe how resource groups behave atdifferent cluster events. It is important to keep in mind the difference between fallover and fallback.These terms appear frequently in discussion of the various resource group policies.


Networks and resource groupsA service IP label can be included in any nonconcurrent resource group - that resource group could haveany of the allowed startup policies except Online on All Available Nodes.

PowerHA SystemMirror cluster configurationsThis chapter provides examples of the types of cluster configurations supported by the PowerHASystemMirror software.

This list is by no means an exhaustive catalog of the possible configurations you can define using thePowerHA SystemMirror software. Rather, use them as a starting point for thinking about the clusterconfiguration best suited to your environment.

Standby configurationsStandby configurations are the traditional redundant hardware configurations where one or more standbynodes stand idle or running a less critical application, and will takeover the critical application should aserver or primary node fail, waiting for a server node to leave the cluster.

Concurrent resource groups are activated on all nodes concurrently and therefore cannot be used in astandby configuration.

Example: Standby configurations with online on first available node startup policy

Figure 2. One-for-one standby configuration where IP label returns to the home node

In this setup, the cluster resources are defined as part of a single resource group. A node list is thendefined as consisting of two nodes. The first node, Node A, is assigned a takeover (ownership) priority of1. The second node, Node B, is assigned a takeover priority of 2.

At cluster startup, Node A (which has a priority of 1) assumes ownership of the resource group. Node A isthe "server" node. Node B (which has a priority of 2) stands idle, ready should Node A fail or leave thecluster. Node B is, in effect, the "standby".

If the server node leaves the cluster, the standby node assumes control of the resource groups owned bythe server, starts the highly available applications, and services clients. The standby node remains activeuntil the home node rejoins the cluster (based on the fallback policy configured). At that point, thestandby node releases the resource groups it has taken over, and the server node reclaims them. Thestandby node then returns to an idle state.

Extending standby configurations

The standby configuration from the previously described example can be easily extended to largerclusters. The advantage of this configuration is that it makes better use of the hardware. Thedisadvantage is that the cluster can suffer severe performance degradation if more than one server nodeleaves the cluster.


The following figure illustrates a three-node standby configuration using the resource groups with thesepolicies:

• Startup policy: Online on First Available Node• Fallover policy: Fallover to Next Priority Node in the List• Fallback policy: Fallback to Home Node

Figure 3. One-for-two standby configuration with three resource groups

In this configuration, two separate resource groups (A and B) and a separate node list for each resourcegroup exist. The node list for Resource Group A consists of Node A and Node C. Node A has a takeoverpriority of 1, while Node C has a takeover priority of 2. The node list for Resource Group B consists ofNode B and Node C. Node B has a takeover priority of 1; Node C again has a takeover priority of 2. (Aresource group can be owned by only a single node in a nonconcurrent configuration.)

Since each resource group has a different node at the head of its node list, the cluster's workload isdivided, or partitioned, between these two resource groups. Both resource groups, however, have thesame node as the standby in their node lists. If either server node leaves the cluster, the standby nodeassumes control of that server node's resource group and functions as the departed node.

In this example, the standby node has three network interfaces (not shown) and separate physicalconnections to each server node's external disk. Therefore, the standby node can, if necessary, take overfor both server nodes concurrently. The cluster's performance, however, would most likely degrade whilethe standby node was functioning as both server nodes.

Takeover configurationsIn the takeover configurations, all cluster nodes do useful work, processing part of the cluster's workload.There are no standby nodes. Takeover configurations use hardware resources more efficiently thanstandby configurations since there is no idle processor. Performance can degrade after node failure,however, since the load on remaining nodes increases.

One-sided takeoverThis configuration has two nodes actively processing work, but only one node providing highly availableservices to cluster clients. That is, although there are two sets of resources within the cluster (forexample, two server applications that handle client requests), only one set of resources needs to behighly available.

The following figure illustrates a two-node, one-sided takeover configuration. In the figure, a lowernumber indicates a higher priority.


Figure 4. One-sided takeover configuration with resource groups in which IP label returns to the home node

This set of resources is defined as a PowerHA SystemMirror resource group and has a node list thatincludes both nodes. The second set of resources is not defined as a resource group and, therefore, is nothighly available.

At cluster startup, Node A (which has a priority of 1) assumes ownership of Resource Group A. Node A, ineffect, “owns” Resource Group A. Node B (which has a priority of 2 for Resource Group A) processes itsown workload independently of this resource group.

If Node A leaves the cluster, Node B takes control of the shared resources. When Node A rejoins thecluster, Node B releases the shared resources.

If Node B leaves the cluster, however, Node A does not take over any of its resources, since Node B'sresources are not defined as part of a highly available resource group in whose chain this nodeparticipates.

This configuration is appropriate when a single node is able to run all the critical applications that need tobe highly available to cluster clients.

Mutual takeoverThe mutual takeover for nonconcurrent access configuration has multiple nodes, each of which providesdistinct highly available services to cluster clients. For example, each node might run its own instance of adatabase and access its own disk.

Furthermore, each node has takeover capacity. If a node leaves the cluster, a surviving node takes overthe resource groups owned by the departed node.

The mutual takeover for nonconcurrent access configuration is appropriate when each node in the clusteris running critical applications that need to be highly available and when each processor is able to handlethe load of more than one node.

The following figure illustrates a two-node mutual takeover configuration for nonconcurrent access. Inthe figure, a lower number indicates a higher priority.


Figure 5. Mutual takeover configuration for nonconcurrent access

The key feature of this configuration is that the cluster's workload is divided, or partitioned, between thenodes. Two resource groups exist, in addition to a separate resource chain for each resource group. Thenodes that participate in the resource chains are the same. It is the differing priorities within the chainsthat designate this configuration as mutual takeover.

The chains for both resource groups consist of Node A and Node B. For Resource Group A, Node A has atakeover priority of 1 and Node B has a takeover priority of 2. For Resource Group B, the takeoverpriorities are reversed. Here, Node B has a takeover priority of 1 and Node A has a takeover priority of 2.

At cluster startup, Node A assumes ownership of the Resource Group A, while Node B assumesownership of Resource Group B.

If either node leaves the cluster, its peer node takes control of the departed node's resource group. Whenthe "owner" node for that resource group rejoins the cluster, the takeover node relinquishes theassociated resources; they are reacquired by the integrating home node.

Two-node mutual takeover configurationIn this configuration, both nodes have simultaneous access to the shared disks and own the same diskresources.

The following figure illustrates a two-node mutual takeover configuration for concurrent access:


Figure 6. Two-node mutual takeover configuration for concurrent access

In this example, both nodes are running an instance of a server application that accesses the database onthe shared disk. The application's proprietary locking model is used to arbitrate application requests fordisk resources.

Running multiple instances of the same server application allows the cluster to distribute the processingload. As the load increases, additional nodes can be added to further distribute the load.


Planning for PowerHA SystemMirror for LinuxAll the relevant Red Hat Package Manager (RPM) are installed automatically on starting the PowerHASystemMirror installation script.

To install the PowerHA SystemMirror for Linux, the following RPMs are used internally:

• powerhasystemmirror• powerhasystemmirror.adapter• powerhasystemmirror.policies• powerhasystemmirror.policies.one• powerhasystemmirror.policies.two• powerhasystemmirror.sappolicy

PowerHA SystemMirror internally uses Reliable Scalable Cluster Technology (RSCT) for the clusteringtechnology. The required versions of the RSCT RPM are installed automatically by default when you installthe PowerHA SystemMirror. During installation of the PowerHA SystemMirror, if RSCT is detected, and thelevel of RSCT is lower than the required RSCT package, then the currently installed RSCT package isupgraded. PowerHA SystemMirror for Linux installs the following RSCT RPMs:

• rsct.basic• rsct.core• rsct.core.utils• rsct.opt.storagerm• src

• If you define the disk tiebreaker resources, the disk on which IBM. TieBreaker resources are storedmust not be used to store file systems.

• Internet Protocol version 6 (IPv6) configuration is not supported in the PowerHA SystemMirror forLinux.

• You can check the firewall status by running the systemctl status firewalld.servicecommand. Firewall must be disabled or you must open the below ports:

657/tcp16191/tcp657/udp12143/udp12347/udp 12348/udp

When a node is configured with multiple connections to a single network, the network interfaces servedifferent functions in the PowerHA SystemMirror.

A service interface is a network interface that is configured with the PowerHA SystemMirror service IPlabel. The service IP label is used by clients to access application programs. The service IP is onlyavailable when the corresponding resource group is online.

A persistent node IP label is an IP alias that can be assigned to a specific node on a cluster network. Apersistent node IP label always stays on the same node (node-bound), and coexists on an NIC thatalready has a service or boot IP label defined. A persistent node IP label does not require installing anextra physical NIC on that node.

If you assign a persistent node IP label, it provides a node-bound address that you can use foradministrative purposes because a connection to a persistent node IP label always goes to a specificnode in the cluster. You can have one persistent node IP label per node.

For PowerHA SystemMirror, you must configure a persistent IP label for each cluster node. This is usefulto access a particular node in a PowerHA SystemMirror cluster for running reports or for diagnostics. This


https://www.ibm.com/support/knowledgecenter/en/SGVKBA

provides advantage that the PowerHA SystemMirror can access the persistent IP label on the nodedespite individual NIC failures, provided spare NICs are present on the network.

If you assign IP aliases to NICs, it allows you to create more than one IP label on the same networkinterface. During an IP address takeover by using the IP aliases, when an IP label moves from one NIC toanother, the target NIC receives the new IP label as an IP alias and keeps the original IP label andhardware address.

Configuring networks for IP address takeover (IPAT) by using IP aliases simplifies the networkconfiguration in the PowerHA SystemMirror. You can configure a service address and one or more bootaddresses for NICs.

PowerHA SystemMirror uses a technology referred to as IPAT by using IP aliases for keeping IP addresseshighly available.

If you are planning for IP address takeover by using IP aliases, review the following information:

• Each network interface must have a boot IP label that is defined in the PowerHA SystemMirror. Theinterfaces that are defined in the PowerHA SystemMirror are used to keep the service IP addresseshighly available.

• The following subnet requirements apply if multiple interfaces are present on a node that is attached tothe same network:

– All boot addresses must be defined on different subnets.– Service addresses must be on a different subnet from all boot addresses and persistent addresses.

• Service address labels that are configured for IP address takeover by using IP aliases can be included inall nonconcurrent resource groups.

• The netmask for all IP labels in a PowerHA SystemMirror for Linux network must be the same.

During a node fallover event, the service IP label that is moved is placed as an alias on the NIC of targetnode in addition to any other service labels configured on that NIC.

If your environment has multiple adapters on the same subnet, all the adapters must have the samenetwork configuration and the adapters must be part of the PowerHA SystemMirr

powerha systemmirror for linux version 7 · powerha systemmirror gui installation process from the...

Documents