sun cluster automated test environment 1.2 private ...sun cluster automated test environment 1.2...

Sun Cluster Automated TestEnvironment 1.2 Private

Interconnect Qualification User’sGuide

Sun Microsystems, Inc.4150 Network CircleSanta Clara, CA 95054U.S.A.

Part No: 816–7555–10October 2002, Revision A

Click t

o BUY NOW!PDF-XChange Editor

ww

w.tracker-software.c

om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com

https://www.tracker-software.com/product/pdf-xchange-editor




Copyright 2003 Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 U.S.A. All rights reserved.

This product or document is protected by copyright and distributed under licenses restricting its use, copying, distribution, and decompilation. Nopart of this product or document may be reproduced in any form by any means without prior written authorization of Sun and its licensors, if any.Third-party software, including font technology, is copyrighted and licensed from Sun suppliers.

Parts of the product may be derived from Berkeley BSD systems, licensed from the University of California. UNIX is a registered trademark in the U.S.and other countries, exclusively licensed through X/Open Company, Ltd.

Sun, Sun Microsystems, the Sun logo, docs.sun.com, AnswerBook, AnswerBook2, and Solaris are trademarks, registered trademarks, or service marksof Sun Microsystems, Inc. in the U.S. and other countries. All SPARC trademarks are used under license and are trademarks or registered trademarksof SPARC International, Inc. in the U.S. and other countries. Products bearing SPARC trademarks are based upon an architecture developed by SunMicrosystems, Inc.

The OPEN LOOK and Sun™ Graphical User Interface was developed by Sun Microsystems, Inc. for its users and licensees. Sun acknowledges thepioneering efforts of Xerox in researching and developing the concept of visual or graphical user interfaces for the computer industry. Sun holds anon-exclusive license from Xerox to the Xerox Graphical User Interface, which license also covers Sun’s licensees who implement OPEN LOOK GUIsand otherwise comply with Sun’s written license agreements.

Federal Acquisitions: Commercial Software–Government Users Subject to Standard License Terms and Conditions.

DOCUMENTATION IS PROVIDED “AS IS” AND ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES,INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE OR NON-INFRINGEMENT, AREDISCLAIMED, EXCEPT TO THE EXTENT THAT SUCH DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

Copyright 2003 Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 U.S.A. Tous droits réservés

Ce produit ou document est protégé par un copyright et distribué avec des licences qui en restreignent l’utilisation, la copie, la distribution, et ladécompilation. Aucune partie de ce produit ou document ne peut être reproduite sous aucune forme, par quelque moyen que ce soit, sansl’autorisation préalable et écrite de Sun et de ses bailleurs de licence, s’il y en a. Le logiciel détenu par des tiers, et qui comprend la technologie relativeaux polices de caractères, est protégé par un copyright et licencié par des fournisseurs de Sun.

Des parties de ce produit pourront être dérivées du système Berkeley BSD licenciés par l’Université de Californie. UNIX est une marque déposée auxEtats-Unis et dans d’autres pays et licenciée exclusivement par X/Open Company, Ltd.

Sun, Sun Microsystems, le logo Sun, docs.sun.com, AnswerBook, AnswerBook2, et Solaris sont des marques de fabrique ou des marques déposées, oumarques de service, de Sun Microsystems, Inc. aux Etats-Unis et dans d’autres pays. Toutes les marques SPARC sont utilisées sous licence et sont desmarques de fabrique ou des marques déposées de SPARC International, Inc. aux Etats-Unis et dans d’autres pays. Les produits portant les marquesSPARC sont basés sur une architecture développée par Sun Microsystems, Inc.

L’interface d’utilisation graphique OPEN LOOK et Sun™a été développée par Sun Microsystems, Inc. pour ses utilisateurs et licenciés. Sun reconnaîtles efforts de pionniers de Xerox pour la recherche et le développement du concept des interfaces d’utilisation visuelle ou graphique pour l’industriede l’informatique. Sun détient une licence non exclusive de Xerox sur l’interface d’utilisation graphique Xerox, cette licence couvrant également leslicenciés de Sun qui mettent en place l’interface d’utilisation graphique OPEN LOOK et qui en outre se conforment aux licences écrites de Sun.

CETTE PUBLICATION EST FOURNIE “EN L’ETAT” ET AUCUNE GARANTIE, EXPRESSE OU IMPLICITE, N’EST ACCORDEE, Y COMPRIS DESGARANTIES CONCERNANT LA VALEUR MARCHANDE, L’APTITUDE DE LA PUBLICATION A REPONDRE A UNE UTILISATIONPARTICULIERE, OU LE FAIT QU’ELLE NE SOIT PAS CONTREFAISANTE DE PRODUIT DE TIERS. CE DENI DE GARANTIE NES’APPLIQUERAIT PAS, DANS LA MESURE OU IL SERAIT TENU JURIDIQUEMENT NUL ET NON AVENU.

021013@4660

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Contents

Preface 7

1 Qualifying Your Private Interconnect 11

2 Setting Up the Sun Cluster Automated Test Environment Software 13

3 Installing the Sun Cluster Automated Test Environment Software 15

Installing the Sun Cluster Automated Test Environment Software 15� How to Install the SCATE Packages 16

4 Sun Cluster Automated Test Environment Software Configuration Requirements19

Hardware Configuration Requirements for the Sun Cluster Automated Tests 19Node Requirements 20SCATE Server (system) Requirements 20Client Requirements 20

Consolidated Test Configuration Requirements for the Sun Cluster Automated Tests21Individual Test Configuration Requirements for the Sun Cluster Automated Tests22

Sun Cluster Functional Tests 22Cluster File System Functional Tests 22Cluster Interconnect Functional Tests 23Global Device Functional Tests 23Oracle Parallel Server/Real Application Clusters Functional Tests 23

3

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Scalable Service Functional Tests 23Sun Cluster Load Tests 24

Global Device Load Tests 24Scalable Service Load Tests 24Sun Cluster HA for NFS Load Tests 24Sun Cluster HA for Apache/Sun Cluster HA for Sun ONE Web Server LoadTests 24

Sun Cluster Load Tests With Faults 25

5 Identifying the Sun Cluster Automated Tests That Your Qualification Requires27

6 Using the Sun Cluster Automated Test Environment Software 29

Starting the SCATE Server (process) 29� How to Start the SCATE Server (process) 30

Launching the Sun Cluster Automated Test Environment Software User Interface30

� How to Launch the Sun Cluster Automated Test Environment Software UserInterface 30

Running the Sun Cluster Automated Tests 31� How to Use the SCATE User Interface to Run the Sun Cluster AutomatedTests 31� How to Use a Test Request File to Run the Sun Cluster Automated Tests 32

Tuning the Sun Cluster HA for NFS Load Tests 32� How to Tune the Sun Cluster HA for NFS Load Tests 33

Administering the Sun Cluster Automated Tests 36� How to Obtain Test Status Information 36� How to Recover From Test Suites That Exit Prematurely 36

Retrieving Test Results 37Report of Test Results 37Summary of Test Results 37Details of Test Results 38

Generating a Test Results File 38How to Generate a Test Results File 38

Analyzing Test Results 39

7 Using the Sun Cluster System Fault Generator 41

Overview 41

4 Sun Cluster Automated Test Environment 1.2 Private Interconnect Qualification User’s Guide • October 2002, RevisionA

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Using the Sun Cluster Fault Client (process) Interface 42Implementing Existing Faults and Events 43

Plugins, Faults, and Events 43Fault and Event Syntax 44Fault Client Configuration File Keywords 45

Generating a Fault Client Configuration File Automatically 46Generating a Summary of Fault and Event Results 46

Example—Summary of Fault and Event Results 47Creating Customized Faults and Events 48

Sun Cluster Fault Client (process) Interface 48Sun Cluster Fault Server (process) Interface 51

Example—Developing a Fault Plugin Module 53

A Understanding the Sun Cluster Automated Test Environment SoftwareFramework 55

Sun Cluster Automated Test Environment Software Framework Features 55Sun Cluster Automated Test Environment Software Framework Components 56

scaterun Utility 58SCATE Server (process) 58Sun Explorer Data Collector 58Sun Cluster System Fault Generator 58

Sun Cluster Fault Client (process) 59Sun Cluster Fault Server (process) 59

SCATE Server (system) 59

B Understanding the Sun Cluster Automated Tests 61

Sun Cluster Functional Tests 61Cluster File System Functional Tests 62Cluster Interconnect Functional Tests 62Global Device Functional Tests 62NAFO Functional Tests 62Oracle Parallel Server/Real Application Clusters Functional Tests 63RGM Functional Tests 63Scalable Service Functional Tests 63Solstice DiskSuite Functional Tests 63VERITAS Volume Manager Functional Tests 63

Sun Cluster Load Tests 64

Contents 5

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Sun Cluster HA for NFS Load Tests 64

Global Device Load Tests 64

Scalable Service Load Tests 65

Sun Cluster HA for Apache/Sun Cluster HA for Sun ONE Web Server LoadTests 65

Sun Cluster HA for Oracle Load Tests 65

Sun Cluster Load Tests With Faults 65

Glossary 67

Index 69


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Preface

This document provides the information that you need to understand, install, and runthe Sun™ Cluster Automated Test Environment (SCATE). You require SCATE toqualify your private interconnect with Sun Cluster software. The audience for thisdocument consists of experienced engineers with Solaris™ and Sun Cluster softwareadministration and hardware experience.

Using UNIX CommandsThis document might not contain complete information on basic UNIX® commandsand procedures such as shutting down the system, booting the system, andconfiguring devices.

See one or more of the following for this information.

� Online documentation for the Solaris™ software environment� Other software documentation that you received with your system� Solaris operating environment man pages

7

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Typographic Conventions

Typeface or Symbol Meaning Examples

AaBbCc123 The names of commands, files, anddirectories; on-screen computeroutput

Edit your .login file.

Use ls -a to list all files.

% You have mail.

AaBbCc123 What you type, when contrastedwith on-screen computer output

% su

Password:

AaBbCc123 Book titles, new words or terms,words to be emphasized

Read Chapter 6 in the User’s Guide.

These are called class options.

You must be superuser to do this.

Command-line variable; replacewith a real name or value

To delete a file, type rm filename.

Shell Prompts

Shell Prompt

C shell machine_name%

C shell superuser machine_name#

Bourne shell and Korn shell $

Bourne shell and Korn shell superuser #


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Related Documentation

Application Title Part Number

Data Service Qualification User’sGuide

Sun Cluster Automated TestEnvironment 1.2 Data ServiceQualification User’s Guide

816-7558

Private Interconnect QualificationUser’s Guide

Sun Cluster Automated TestEnvironment 1.2 Private InterconnectQualification User’s Guide

816-7555

Public Network Qualification User’sGuide

Sun Cluster Automated TestEnvironment 1.2 Public NetworkQualification User’s Guide

816-7556

Release Notes Sun Cluster Automated TestEnvironment 1.2 Release Notes

816-7559

Server Platform Qualification User’sGuide

Sun Cluster Automated TestEnvironment 1.2 Server PlatformQualification User’s Guide

816-7557

Storage Qualification User’s Guide Sun Cluster Automated TestEnvironment 1.2 Storage QualificationUser’s Guide

816-7554

Accessing Sun Documentation OnlineThe docs.sun.comSM web site enables you to access a select group of Sun technicaldocumentation on the Web. You can browse the docs.sun.com archive or search fora specific book title or subject at:

http://docs.sun.com

Preface 9

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Getting HelpIf you have problems installing or using Sun Cluster, contact your service providerand provide the following information:

� Your name and email address (if available)

� Your company name, address, and phone number

� The model number and serial number of your systems

� The release number of the operating environment (for example, the Solaris 8release)

� The release number of Sun Cluster (for example, the Sun Cluster 3.0 release)

Use the following commands to gather information on your system for your serviceprovider.

Command Function

prtconf -v Displays the size of the system memory and reportsinformation about peripheral devices

psrinfo -v Displays information about processors

showrev -p Reports which patches are installed

prtdiag -v Displays system diagnostic information

/usr/cluster/bin/scinstall-pv

Displays Sun Cluster release and package versioninformation

Also have available the contents of the /var/adm/messages file.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





CHAPTER 1

Qualifying Your Private Interconnect

This chapter provides information on how to qualify your private interconnect.

Table 1–1 lists the tasks that you must perform to qualify your private interconnect onthe Sun Cluster software. Perform these tasks in the order that they are listed.

TABLE 1–1 Task Map: Qualifying Your Hardware

Task For Instructions

Review the Private Interconnect Qualification Plan Your Sun Cluster representative

Set up your Sun Cluster Automated Test Environmentsoftware if you have not already done so.

Chapter 2

Identify the Sun Cluster Automated Tests yourqualification requires.

Chapter 5

Run the Sun Cluster Functional Tests, the Sun ClusterLoad Tests, and the Sun Cluster Load Tests With Faults.You must pass these tests before you proceed.

“Running the Sun ClusterAutomated Tests” on page 31

Partner with your Sun Cluster representative to developmanual tests.

Your Sun Cluster representative

Perform the manual tests. You must pass these testsbefore you proceed.

Your Sun Cluster representative

Request certification from Sun Cluster. Your Sun Cluster representative

11

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





CHAPTER 2

Setting Up the Sun Cluster AutomatedTest Environment Software

This chapter provides information on how to install the Sun Cluster Automated TestEnvironment software (SCATE).

Table 2–1 lists the tasks that you must perform to install the Sun Cluster AutomatedTest Environment software (SCATE). Perform these tasks in the order that they arelisted.

TABLE 2–1 Task Map: Setting Up the Sun Cluster Automated Test Environment software

Task For Instructions

Plan for the SCATE configuration requirements. Chapter 4

Install the Sun Cluster hardware. Ensure that thehardware meets the SCATE node and diskrequirements.

Sun Cluster 3.0 12/01 Hardware Guide

Install the Solaris operating environment and SunCluster software.

Sun Cluster 3.0 12/01 SoftwareInstallation Guide

Create the metadevices or volumes needed to meet theSCATE metadevice or volume requirements.

Chapter 4

Update Network Time Protocol (NTP) or outside timesource if the time between all nodes is notsynchronized.

Sun Cluster 3.0 12/01 SoftwareInstallation Guide

Install the Sun Cluster Automated Test Environmentsoftware.

“Installing the Sun ClusterAutomated Test EnvironmentSoftware” on page 15

Start the SCATE Server (process). “Starting the SCATE Server(process)” on page 29

13

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





CHAPTER 3

Installing the Sun Cluster AutomatedTest Environment Software

This chapter provides the procedure on how to install the Sun Cluster Automated TestEnvironment software (SCATE).

Installing the Sun Cluster AutomatedTest Environment SoftwareThis section contains the procedure on how to install the Sun Cluster Automated TestEnvironment software (SCATE) packages. Figure 3–1 illustrates this installationprocedure. This procedure assumes that you installed Sun Cluster software andhardware.

15

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





SCATE Server

SUNWsctsSUNWfltc

SUNWpl5u

Client

Client 1

SUNWsctsSUNWpldbiSUNWpl5u

ClientSystems

Node 1

SUNWsctsSUNWflts

SUNWexploSUNWpldbiSUNWpl5u

Public Network

Node 2

SUNWsctsSUNWflts


Node N

SUNWsctsSUNWflts


Multihostdisks

FIGURE 3–1 Installing the Sun Cluster Automated Test Environment software Packages

� How to Install the SCATE PackagesUse this procedure to install the SCATE packages by using the scatesetup utility orby manually installing the packages as shown in Figure 3–1.

1. Verify that the /.rhosts file resides on all nodes and client machines and enablesaccess from the SCATE Server (system).

2. How do you want to install the SCATE packages?

� If manually, skip to Step 5� If automatically by using the scatesetup utility, proceed to Step 3

3. Change directory to the top level of the SCATE distribution image and startscatesetup.

# ./scatesetup

4. Choose the menu item that corresponds to your desired installation procedure, andthen skip to Step 9.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





5. On the SCATE Server (system) and from the package location, install the SunCluster Automated Test Environment software and the Sun Cluster Fault Client(process) packages.

# pkgadd -d . SUNWscts SUNWfltc

6. Determine which Perl Database Interface package you must install. You will installthis package in Step 7.

Your choice is based on the Solaris operating environment and the Oracle softwareversion you want to test. There are four versions of this package, one for eachcombination of Solaris 8, Solaris 9, Oracle8, and Oracle9i.

� Solaris 8 software with Oracle8 software� Solaris 8 software with Oracle9i software� Solaris 9 software with Oracle8 software� Solaris 9 software with Oracle9i software

7. On all clients and from the package location, install the Perl Database Interfacepackage you identified in Step 6 and the Sun Cluster Automated Test Environmentsoftware package.

# pkgadd -d . SUNWpldbi SUNWscts

8. On all nodes and from the package location, install the Sun Cluster Fault Server(process) package, the Sun Cluster Automated Test Environment software package,and the Perl Database Interface package you identified in Step 6. Install thesepackages in the following order.

# pkgadd -d . SUNWflts SUNWscts SUNWpldbi

9. On all nodes and from the package location, install the Sun Explorer Data Collector.

# pkgadd -d . SUNWexplo

You cannot use the scatesetup utility to install the Sun Explorer Data Collectorpackage because the installation is interactive.

10. Do you have the Perl package for the Solaris operating environment installed on allclients, all nodes, and the SCATE Server (system)?

# pkginfo SUNWpl5u

This Perl package is not included in the SCATE release. This Perl package ships withSolaris 8, at minimum, release. The Sun Cluster System Fault Generator depends onthis package.

� If yes, skip to Step 13� If no, proceed to Step 11

11. Locate the Perl package for Solaris on the Solaris installation CD.

# cd cd-image/Solaris_version/Product

12. On all clients, all nodes, and the SCATE Server (system), install the Perl package forSolaris.

Chapter 3 • Installing the Sun Cluster Automated Test Environment Software 17

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





# pkgadd -d . SUNWpl5u

This package must be installed on all clients because Sun Cluster HA for Oracle LoadTests are written in Perl. The package must also be installed on all nodes and theSCATE Server (system) because the Sun Cluster System Fault Generator isimplemented in Perl.

13. Verify that you installed all packages correctly.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





CHAPTER 4

Sun Cluster Automated TestEnvironment Software ConfigurationRequirements

This chapter provides the configuration requirements that you must satisfy toconfigure the Sun Cluster Automated Test Environment software (SCATE).

Before you can run the Sun Cluster Automated Tests, you must satisfy the hardwareconfiguration requirements. To run all the Sun Cluster Automated Tests required foryour qualification, you must meet the consolidated test requirements. To run the SunCluster Automated Tests individually, you must meet the individual test configurationrequirements.

This chapter consists of the following sections.

� “Hardware Configuration Requirements for the Sun Cluster Automated Tests”on page 19

� “Consolidated Test Configuration Requirements for the Sun Cluster AutomatedTests” on page 21

� “Individual Test Configuration Requirements for the Sun Cluster Automated Tests”on page 22

Hardware Configuration Requirementsfor the Sun Cluster Automated TestsThis section provides the hardware configuration requirements for all Sun ClusterAutomated Tests. Before you can run the Sun Cluster Automated Tests, you mustsatisfy all hardware configuration requirements outlined in this section. Theserequirements are in addition to the requirements that are outlined in the existing SunCluster user documentation.

19

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Node RequirementsYour nodes must meet the following requirements.

� Platform that the Sun Cluster software supports.� Minimum 150 Mbytes of free disk space on the partition that hosts the /opt file

system.� Remote shell (rsh) access to all clients and the SCATE Server (system) as user

scate and user root. Set up the .rhosts files to allow for rsh access.� Network Time Protocol (NTP) configured.

Note – You must synchronize the time between all nodes, whether you use NTP oranother time source. If you do not synchronize the time between nodes, some ofthe Sun Cluster Automated Tests can fail. For the procedure on how to update theNTP, see the Sun Cluster 3.0 12/01 Software Installation Guide.

� Default login shell for user root on the SCATE Server (system) must becompatible with Bourne shell. If you are not sure whether root’s shell iscompatible, set it to /bin/sh to ensure proper operation.

SCATE Server (system) RequirementsYour SCATE Server (system) must meet the following requirements.

� One Sun server.� Solaris 8 software, at minimum, must be installed.� Minimum 150 Mbytes of free disk space on the partition that hosts the /opt file

system.� Remote shell (rsh) access to all clients and all nodes as user scate and user root.

Set up the .rhosts files to allow for rsh access.� Default login shell for user root on the SCATE Server (system) must be

compatible with Bourne shell. If you are not sure whether root’s shell iscompatible, set it to /bin/sh to ensure proper operation.

Client RequirementsYour Client Machine must meet the following requirements.

� Solaris 8 software, at minimum, must be installed.� Remote shell (rsh) access to the SCATE Server (system) and all nodes as user

scate and user root. Set up the .rhosts files to allow for rsh access.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





� Minimum 150 Mbytes of free disk space on the partition that hosts the /opt filesystem.

� Minimum one IP address on the same subnet as the failover IP address or theshared IP address on the cluster.

� Default login shell for user root on the SCATE Server (system) must becompatible with Bourne shell. If you are not sure whether root’s shell iscompatible, set it to /bin/sh to ensure proper operation.

Consolidated Test ConfigurationRequirements for the Sun ClusterAutomated TestsThis section combines the individual test configuration requirements for all SunCluster Automated Tests.

TABLE 4–1 Consolidated Test Configuration Requirements for the Sun Cluster AutomatedTests

Requirement Type Number/Size/Type Instructions/Notes

Node and Client Requirements

IP addresses 2 None

Software Wget software Wget software must be installed onall client machines. This softwaremust be installed in the samelocation on each client machine. Youcan download Wget software fromhttp://www.sunfreeware.com. Wgetsoftware also resides on the Solaris 86/00, at minimum, CompanionCD-ROM. The package name isSFWwget.

Device Requirements

Chapter 4 • Sun Cluster Automated Test Environment Software Configuration Requirements 21

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





TABLE 4–1 Consolidated Test Configuration Requirements for the Sun Cluster AutomatedTests (Continued)Metadevices andVolumes

� 2 Solstice DiskSuite/SolarisVolume Managermetadevicesor

� 2 VxVM volumes� These devices must each

have 2 Gbytes of free diskspace

Metadevices or volumes must bepart of independent device services.

Volumes and LUNs � 2 control file devices� 2 data file devices� 2 log files for each node� Devices must be at least 400

MBytes in size.

You need these volumes and LUNsto run the Oracle ParallelServer/Real Application ClustersFunctional Tests.

VxVM volumes must be in a sharedCVM diskgroup or a virtual LUNfrom a hardware RAID storagedevice.

Individual Test ConfigurationRequirements for the Sun ClusterAutomated TestsThis section provides the individual test configuration requirements for all Sun ClusterAutomated Tests.

Sun Cluster Functional TestsThe Sun Cluster Functional Tests are as follows.

Cluster File System Functional TestsThis test suite has the following requirements.

� A minimum of two Solstice DiskSuite/Solaris Volume Manager metadevices orVxVM volumes.

� These two Solstice DiskSuite/Solaris Volume Manager metadevices or VxVMvolumes must be part of two independent device services.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





� These two Solstice DiskSuite/Solaris Volume Manager metadevices or VxVMvolumes must have a minimum of 10 Mbytes of free disk space.

Cluster Interconnect Functional TestsThis test suite does not have any requirements.

Global Device Functional Tests� Minimum of two Solstice DiskSuite/Solaris Volume Manager metadevices or

VxVM volumes.� These two Solstice DiskSuite/Solaris Volume Manager metadevices or VxVM

volumes must be part of two independent device services.� These two Solstice DiskSuite/Solaris Volume Manager metadevices or VxVM

volumes must also have a minimum of 10 Mbytes of free disk space.

Oracle Parallel Server/Real Application ClustersFunctional TestsThis test suite has the following requirements.

� A supported version of OPS/RAC must be installed locally on each node. For moredetails on installing OPS/RAC in the Sun Cluster environment, see the SunCluster 3.0 12/01 Data Services Installation and Configuration Guide.

� Client machines with access to the Oracle client libraries. These libraries can eitherbe installed locally on each client or once on an NFS server.

� An oracle user account on each client machine and all nodes. This oracle useraccount is the account that is used for database administration. This account isusually created as part of the Oracle installation process. This account must allowrhost permissions for itself and user scate.

� The following volumes.

Two control file devices

Two data file devices

Two log files for each node

These volumes must be online. These volumes can either be VERITAS volumes in ashared diskgroup or virtual LUNs from a hardware RAID storage device. All ofthese devices must be at least 400 MBytes in size.

Scalable Service Functional TestsThis test suite has the following requirements.

� One IP address


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





� One external client machine

Sun Cluster Load TestsThe Sun Cluster Load Tests are as follows.

Note – The Sun Cluster Load Tests include load generators that are flexible. Theseload generators run on a wide range of cluster configurations. The Sun Cluster LoadTests provide a set of tunable parameters. These parameters include number of usersand number of client machines. Tune these parameters for your cluster configurationto ensure adequate utilization of CPU, memory, and network resources.

Global Device Load TestsThis test suite has the following requirements.

� Minimum two Solstice DiskSuite/Solaris Volume Manager metadevices or VxVMvolumes.

� These two Solstice DiskSuite/Solaris Volume Manager metadevices or VxVMvolumes must be part of two independent device services.

� These two Solstice DiskSuite/Solaris Volume Manager metadevices or VxVMvolumes must have a minimum of 2 Gbytes of free disk space.

Scalable Service Load TestsThis test suite has the following requirements.

� Two IP addresses� External client machines to produce the required load for your configuration

Sun Cluster HA for NFS Load TestsThis test suite has the following requirements.

� Minimum two volumes with 2 Gbytes of free disk space.� Sun Cluster HA for NFS data service must be installed on all nodes.

Sun Cluster HA for Apache/Sun Cluster HA for Sun ONEWeb Server Load TestsThis test suite has the following requirements.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





� Sun Cluster HA for Apache or Sun HA for Sun Open Net Environment (Sun ONE)Web Server must be installed on all nodes.

� Wget software must be installed on all client machines. This software must beinstalled in the same location on each client machine.

Wget software is a utility that provides support for non-interactive download offiles from the internet. You can download Wget software fromhttp://www.sunfreeware.com. Wget software also resides on the Solaris 8 6/00, atminimum, Companion CD-ROM. The package name is SFWwget.

� Two IP addresses.

� One raw disk slice that has a minimum of 2 Gbytes of free disk space.

� External client machines to produce the required load for your configuration.

Sun Cluster Load Tests With FaultsThere are no specific requirements for Sun Cluster Load Tests With Faults. The samerequirements that apply to Sun Cluster Load Tests also apply to Sun Cluster LoadTests With Faults.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





CHAPTER 5

Identifying the Sun Cluster AutomatedTests That Your Qualification Requires

This chapter provides a table to help you identify the Sun Cluster Automated Teststhat your specific qualification requires.

The following table lists the Sun Cluster Automated Tests that each SCATEqualification requires.

TABLE 5–1 Requirement Sun Cluster Automated Tests

Tests Name PrivateInterconnectQualification

Public NetworkQualification

ServerPlatformQualification

StorageQualification

Sun Cluster Functional Tests

Cluster File SystemFunctional Tests

Yes Yes Yes Yes

Cluster InterconnectFunctional Tests

Yes No Yes No

Global Device FunctionalTests

Yes Yes Yes Yes

NAFO Functional Tests No Yes Yes No

Oracle Parallel Server/RealApplication ClustersFunctional Tests

Yes No Yes Yes

RGM Functional Tests No Yes Yes No

Scalable Service FunctionalTests

Yes Yes Yes No

Volume ManagerFunctional Tests

No No No Yes

Sun Cluster Load Tests

27

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





TABLE 5–1 Requirement Sun Cluster Automated Tests (Continued)Global Device Load Tests Yes No Yes Yes

Scalable Service Load Tests Yes Yes Yes

Sun Cluster HA for NFSLoad Tests

Yes Yes Yes Yes

Sun Cluster HA for OracleLoad Tests

No No Yes Yes

Sun Cluster HA forApache/Sun Cluster HAfor Sun ONE Web ServerLoad Tests

Yes Yes Yes No

Sun Cluster Load Tests With Faults

Global Device Load TestsWith Faults

Yes No Yes Yes

Scalable Service Load TestsWith Faults

Yes Yes Yes No

Sun Cluster HA forApache/Sun Cluster HAfor Sun ONE Web ServerLoad Tests With Faults

Yes Yes Yes No

Sun Cluster HA for NFSLoad Tests With Faults

Yes Yes Yes Yes

Sun Cluster HA for OracleLoad Tests With Faults

No No Yes Yes


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





CHAPTER 6

Using the Sun Cluster Automated TestEnvironment Software

This chapter provides procedures and information on how to use the Sun ClusterAutomated Test Environment software (SCATE). This chapter consists of the followingsections.

� “Starting the SCATE Server (process)” on page 29� “Launching the Sun Cluster Automated Test Environment Software User Interface”

on page 30� “Running the Sun Cluster Automated Tests” on page 31� “Tuning the Sun Cluster HA for NFS Load Tests” on page 32� “Administering the Sun Cluster Automated Tests” on page 36� “Retrieving Test Results” on page 37� “Generating a Test Results File” on page 38� “Analyzing Test Results” on page 39

Starting the SCATE Server (process)This section contains the procedure on how to start the SCATE Server (process).

After adding the SCATE packages on the SCATE Server (system), the SCATE Server(process) runs continuously. However, if you reboot the SCATE Server (system), youneed to restart the SCATE Server (process). To launch the SCATE user interface(scaterun utility), the SCATE Server (process) must be running. If you reboot theSCATE Server (system), the SCATE Server (process) stops. You must restart the SCATEServer (process) again to launch the SCATE user interface (scaterun utility).

29

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Tip – Avoid the need to start the SCATE Server (process) manually each time youreboot your SCATE Server (system). Create an rc script on the SCATE Server (system)so that the rc script enables the SCATE Server (process) to start each time you bootthe SCATE Server (system).

� How to Start the SCATE Server (process)Use this procedure to start the SCATE Server (process).

1. Log in to the SCATE Server (system) as superuser.

2. Start the SCATE Server (process) if the SCATE Server (process) is not alreadyrunning.

# pgrep -fl scate

# /opt/SUNWscts/scate/scatestart

Launching the Sun Cluster AutomatedTest Environment Software UserInterfaceThis section contains the procedure on how to launch the Sun Cluster Automated TestEnvironment software user interface, the scaterun utility.

The SUNWscts package automatically creates a user name and default passwordwhen you install the package. The user name is scate, and the default password isscate123. The user name and the password enable you to launch the scaterunutility.

� How to Launch the Sun Cluster Automated TestEnvironment Software User InterfaceUse this procedure to launch the Sun Cluster Automated Test Environment softwareuser interface, the scaterun utility.



Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





For the procedure on how to start the SCATE Server (process), see “Starting theSCATE Server (process)” on page 29.

2. Log in to the SCATE Server (system) if you are not already.

The user name is scate, and the default password is scate123.

3. Start the scaterun utility.

$ /opt/SUNWscts/scate/scaterun

Running the Sun Cluster AutomatedTestsThis section contains procedure on how to run the Sun Cluster Automated Tests.

� How to Use the SCATE User Interface to Run theSun Cluster Automated TestsUse this procedure to use the SCATE user interface (scaterun utility) to run the SunCluster Automated Tests.

1. Satisfy the hardware configuration requirements and the individual testconfiguration requirements.

2. Launch the SCATE user interface (scaterun utility).

The scaterun utility prompts you for the name to use for the Test Request File that isto be generated. The scaterun utility also prompts you for the location where theSCATE test logs must be spooled. For the procedure on how to launch the SCATEinterface, see “Launching the Sun Cluster Automated Test Environment Software UserInterface” on page 30.

3. Specify the Test Request File and test log directory.

4. Specify the name of a node.

The SCATE software needs the name of one of the nodes to connect to the cluster andbootstrap the test environment. You can specify any node in the cluster.

5. Choose the menu option number that corresponds to your qualification.

6. Choose the menu option number that corresponds to the Sun Cluster AutomatedTests that your qualification requires.

7. Specify your Sun Cluster software version.

Chapter 6 • Using the Sun Cluster Automated Test Environment Software 31

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





For information on the Sun Cluster software versions that SCATE supports, see theSun Cluster 3.0 5/02 Release Notes.

8. Specify the type of file system that is installed on the cluster.

For information on the file systems that Sun Cluster software supports, see the SunCluster 3.0 5/02 Release Notes.

9. Confirm your selections.

Wait for the scaterun utility to complete the test execution before you proceed.

10. Answer the questions when the scaterun utility prompts you.

11. Review the test results in the SCATE_REPORT file to determine whether all the SunCluster Functional Tests passed.

� If all the tests passed, this step completes this procedure.� If any tests failed, fix the problem with those tests. Repeat Step 2 through Step 10.

� How to Use a Test Request File to Run the SunCluster Automated TestsUse this procedure to use a Test Request File to run the Sun Cluster Automated Tests.The scaterun utility prompts you for configuration information. The SCATE savesthe values that you provide to a Test Request File. The SCATE also prompts you for afile name for this Test Request File. You can use this Test Request File to run the sametests without answering the configuration questions again.


For the procedure on how to start the SCATE Server (process), see “Starting theSCATE Server (process)” on page 29.

2. From the SCATE Server (system), run the Test Request File.

$ scaterun -f test-request-filename

Tuning the Sun Cluster HA for NFSLoad TestsThis section contains the procedure on how to tune the Sun Cluster HA for NFS LoadTests.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





The Sun Cluster HA for NFS Load Tests provide a Load Generator that is flexibleenough to run on a wide range of node and storage hardware. However, as a result ofthis flexibility, you need to tune this tool for each cluster you want to test. Tuningensures maximum utilization of the storage hardware without destabilizing thecluster.

� How to Tune the Sun Cluster HA for NFS LoadTestsUse this procedure to tune the Sun Cluster HA for NFS Load Tests.

Tip – Do not overload the nodes and client machines while you run the Sun ClusterLoad Tests and Sun Cluster Load Tests With Faults. This practice prevents problemsthat are associated with the exhaustion of CPU, memory, and network resources.

Note – If you change the metadevice or volume configuration of your cluster, youmight need to tune the Sun Cluster HA for NFS Load Tests before you run theSunCluster HA for NFS Load Tests again.

1. Determine which file systems that you intend to run the Sun Cluster HA for NFSLoad Tests on. Determine which client machines that you intend to generate theload for the Sun Cluster HA for NFS Load Tests.

a. Determine which metadevices or volumes these file systems reside on.Determine which disk devices these file systems reside on.

b. Record the paths to these file systems.

2. As root on one node, determine the desfree memory threshold that you are usingfor these tests.

adb -k

3. Does the node run on a 32-bit kernel or a 64-bit kernel?

� If the node is running a 32-bit kernel, issue the following command to adb(1).

*desfree*0t8=D

Record the value that this command returns. Use this value, which is in kilobytes,as the memory threshold for the Sun Cluster HA for NFS Load Tests.

� If the node is running a 64-bit kernel, issue the following command to adb(1).

*desfree*0t8=E


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Record the value that this command returns. Use this value, which is in kilobytes,as the memory threshold for the Sun Cluster HA for NFS Load Tests.

4. Open three separate terminal widows.

5. On the primary node that you identified in Step 1, type the following command inTerminal Window #1.

# /usr/bin/iostat -nMx 5

Use the output of the iostat(1M) command to track the percentage of time that thevolume is busy handling I/O transactions.


# /usr/bin/mpstat 5

Use the output of the mpstat(1M) command to track the utilization of the CPUs.


# /usr/bin/vmstat -p 5

Use the output of the vmstat(1M) command to track memory utilization.

8. Launch the SCATE user interface (scaterun utility).

For the procedure on how to launch the SCATE, see “Launching the Sun ClusterAutomated Test Environment Software User Interface” on page 30.

9. From the Available Tests menu, choose Sun Cluster HA for NFS Load Tests to runthe Sun Cluster HA for NFS Load Tests.

The scaterun utility prompts you for configuration information.

10. Have you run the Sun Cluster HA for NFS Load Tests before?

� If yes, accept the default value of 25 client processes.

� If no, increase or decrease the number of client processes by increments of 5, as youfollow the guidelines in Table 6–1.

11. The Sun Cluster HA for NFS Load Tests are now running. Monitor the output of thethree commands you ran in Step 4 .

a. Record the iostat command’s value in the %b field for the metadevices orvolumes being tested.

b. Record the mpstat command’s value in the sys field for each CPU.

c. Record the vmstat command’s value for the free field.

Run the Sun Cluster HA for NFS Load Tests until these three values stabilize. The timethat is required for these three values to stabilize depends on your hardwareconfiguration. When these values stabilize, record these values for use in Step 12.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





12. Do the values that you noted in Step 11 meet the following maximum utilizationguidelines in Table 6–1?

� If yes, you reached maximum utilization on the storage that you intend to qualify.This step completes this procedure.

� If no, perform the following actions.

� On all client machines, stop the Sun Cluster HA for NFS Load Tests.

For the procedure on stopping the Sun Cluster HA for NFS Load Tests, see“How to Recover From Test Suites That Exit Prematurely” on page 36.

� Repeat Step 8 and Step 12.

TABLE 6–1 Optimal Load Test Parameters

Command Field Field Description Value Action

iostat %b Percentage oftime themetadevices orvolumes are busy

Less than 100% Increase thenumber of clientprocesses

vmstat free Amount of freememory

Less thandesfreethreshold inStep 3

Decrease thenumber of clientprocesses

mpstat sys Percentage oftime that theCPUs execute inkernel mode

Less than orequal to 98%

Decrease thenumber of clientprocesses

Note – To achieve an optimal set of load test parameters, maximize the iostatcommand’s %b value as much as possible without exceeding the remaining thresholds.

If you increased the number of client processes and the values in Table 6–1 do notchange, a bottleneck might have occurred on the cluster’s public network interface. Ifyou have a bottleneck, you maximized the load test parameters for this cluster. Nofurther adjustment of the number of client processes is necessary.Your inability to change the value might also be because the current set of clientmachines is unable to drive any additional load. The computing power of the clientmachines determines when the client machines reach their capacity before youmaximize the storage. To verify the client machines’ state, log in to the clientmachines. Run the appropriate diagnostic commands. If the client machines reachedtheir capacity, repeat Step 8 and Step 12 and specify additional client machines, ifadditional client machines are available. If no additional client machines are available,then you maximized the load test parameters for this cluster. No further adjustment ofthe number of client processes is necessary.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Administering the Sun ClusterAutomated TestsThis section contains procedures on how to administer the Sun Cluster AutomatedTests. This section includes the following procedures.

� “How to Obtain Test Status Information” on page 36� “How to Recover From Test Suites That Exit Prematurely” on page 36

� How to Obtain Test Status InformationUse this procedure to determine which tests are running. Use this procedure todetermine how long these tests have been running.

1. Log in to the SCATE Server (system).

2. Request test status information from the SCATE Server (process).

$ scatestatus -l

� How to Recover From Test Suites That ExitPrematurelyUse this procedure to terminate the processes associated with test suites that exitprematurely. Any test suite can exit prematurely, but some test suites require manualrecovery. Table 6–2 lists the test suites that require manual recovery. If you do notperform this procedure after one of theses test suites exits prematurely, you are unableto run that test suite again.

1. Log in to the SCATE Server (system) as user scate.

2. Invoke the script that corresponds to the test suite that exited prematurely.

$ script

TABLE 6–2 Table Caption

Test Suite Script

Global Device Load Tests ~scate/tset/dev_stress/bin/cleanup

Scalable Service Load Tests ~scate/tset/net_stress/bin/cleanup


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





TABLE 6–2 Table Caption (Continued)Test Suite Script

Sun Cluster HA for Apache/SunCluster HA for Sun ONE Web ServerLoad Tests

~scate/http_results/http_cleanup.cluster_nodename

Sun Cluster HA for NFS Load Tests ~scate/nfs_results/nfs_cleanup.cluster_nodename

Sun Cluster HA for Oracle Load Tests ~scate/tset_load/dbms/testdb/oracle/oracle_cleanup

Retrieving Test ResultsThis section contains information on how to retrieve test results.

The SCATE stores test results in the following files.

� Test Results File� Test Summary File� Test Details File

Report of Test ResultsThe SCATE stores a report of test results in a Test Results File. Use the information inthe Test Results File to determine if a test suite passed or failed. You can locate theTestResults File on the SCATE Server (system) at test-log-output-directory/date_time_nodename_jobid. The file name if SCATE_REPORT.

Summary of Test ResultsThe SCATE stores a summary of the test results in a Test Summary File. Use theinformation in the Test Summary File to determine which test cases of a particular testsuite passed or failed. You can locate the Test Summary File on the SCATE Server(system). The Test Summary File resides in a subdirectory specific to the test suite youran. The subdirectory is test-log-output-directory/date_time_nodename_jobid. The defaultfile name is summary.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Details of Test ResultsThe SCATE stores detailed test results in the Test Details File. Use the information inthe Test Details File to determine why test cases of a particular test suite failed. You canlocate the Test Details File on the SCATE Server (system). The Test Details File residesin a subdirectory specific to the test suite you ran. The subdirectory isscate~/test-log-output-directory/date_time_nodename_jobid. Thedefault file name is exec.

test-log-output-directory The directory that you specified when thescaterun utility prompted you

date The month, day, and year when you started thescaterun utility

time The time you started the scaterun utility

nodename One of the nodes in the cluster you tested

jobid The job identification number

Generating a Test Results FileThis section contains the procedure on how to generate a Test Results File.

The SCATE generates a Test Results File after all the Sun Cluster Automated Tests yourequested finish. The tests run sequentially. Therefore, some Sun Cluster AutomatedTests finish before others. You might want the results of the completed tests while youwait for the other tests to finish.

For more information on the Test Results File, see “Retrieving Test Results”on page 37.

How to Generate a Test Results FileUse this procedure to obtain the results of the completed tests while the remainingtests are still in progress.

1. Log in to the SCATE Server (system).

2. From the SCATE Server (system), use the report script to generate the Test ResultsFile.

$ report test-log-output-directory/date_time_nodename_jobid


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





The SCATE names the Test Results File SCATE_REPORT.

test-log-output-directory The directory that you specified when youstarted the scaterun utility

date The month, day, and year that you started thescaterun utility

time The time you started the scaterun utility

nodename One of the nodes in the cluster you tested

jobid The job identification number

Analyzing Test ResultsYou need to understand why your test failed. For information on analyzing testresults, see your Sun service provider.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





CHAPTER 7

Using the Sun Cluster System FaultGenerator

This chapter provides procedures and information on how to use the Sun ClusterSystem Fault Generator. This chapter assumes that you installed the Perl package forSolaris. This chapter consists of the following sections.

� “Overview” on page 41� “Using the Sun Cluster Fault Client (process) Interface” on page 42� “Implementing Existing Faults and Events” on page 43� “Generating a Fault Client Configuration File Automatically” on page 46� “Generating a Summary of Fault and Event Results” on page 46� “Creating Customized Faults and Events” on page 48� “Example—Developing a Fault Plugin Module” on page 53

OverviewThe Sun Cluster System Fault Generator is a client-server application with thefollowing features.

� Capability to simulate a number of faults and events on a running cluster.� Facilities to verify proper recovery from a fault or event.� API that provides fault plugins that enable you to define customized faults and

events for the Sun Cluster System Fault Generator to simulate. For moreinformation on the this API, see “Creating Customized Faults and Events”on page 48.

The Sun Cluster System Fault Generator consists of a daemon that is known as theSun Cluster Fault Server (process). The Sun Cluster Fault Server (process) runs on eachnode in the cluster.

41

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





The Sun Cluster Fault Server (process) listens for requests from a Sun Cluster FaultClient (process) that can be started from a machine external to the cluster. The SunCluster Fault Client (process) sends fault or cluster event requests to the Sun ClusterFault Server (process). The Sun Cluster Fault Client (process) then queries the nodes toverify proper recovery from the fault or event.

The Sun Cluster Fault Client (process) takes input from a Fault Client ConfigurationFile that lists the faults that you want to simulate. For information on the format of theFault Client Configuration File, see “Implementing Existing Faults and Events”on page 43.

Using the Sun Cluster Fault Client(process) InterfaceThis section contains information on how to use the Sun Cluster Fault Client (process)interface.

Table 7–1 lists the command line options that the Sun Cluster Fault Client (process)uses. After you install the SCATE packages, you can locate the Sun Cluster FaultClient (process) at /opt/SUNWfltc/bin/faultclient.pl.

TABLE 7–1 Options

Option Description

-f path path: Absolute path to the Fault Client Configuration File

The default is ./faults.conf.

-h hostname hostname: Hostname of one of the nodes to be tested

The Sun Cluster Fault Client (process) connects to this node, and thenqueries the node for information to determine the names of the othernodes. The Sun Cluster Fault Client (process) implements other fault clientbootstrap processes. You can specify any one of the these nodes. Thisoption is required.

-l log-level log-level: Log detail level.

Valid values are 1, 2, or 3, with 3 being the most verbose. The default is1.

-L log-dir log-dir: Directory where the Sun Cluster Fault Client (process) resides

The default log directory is ., which is the current working directory(CWD) of the Sun Cluster Fault Client (process).


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





TABLE 7–1 Options (Continued)Option Description

-n iterations iterations: Number of times that you want to iterate through the fault andevent requests that are defined in the Fault Client Configuration File

The default is 1.

-p port port: Port on which the Sun Cluster Fault Server (process) listens

The default is 9876.

-P plugin-path plugin-path: Optional search path for fault plugins

The default search path is /opt/$PKG/lib, where $PKG is eitherSUNWfltc or SUNWflts. Multiple paths can be specified by separatingmultiple paths with colons.

Implementing Existing Faults andEventsThis section contains information on how to implement existing faults and events.

The faultclient.pl tool uses the Fault Client Configuration File to determine whatfaults or events to request from the Sun Cluster Fault Server (process). Valid entriescan be either be fault and event requests, as defined in a fault plugin, or a fault clientkeyword.

Table 7–2 lists the Sun Cluster System Fault Generator’s fault plugins and the faultsand events that accompany each plugin. Table 7–3 contains the syntax to implementthese faults and events. Table 7–4 provides the keywords to register each plugin.

Plugins, Faults, and EventsThe Sun Cluster System Fault Generator includes the following two plugins toimplement faults and events.

� standard.pm� transports.pm

Chapter 7 • Using the Sun Cluster System Fault Generator 43

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





TABLE 7–2 Plugins, Faults, and Events

Plugin Faults and Events

standard.pm � dg_switchover� fail_pnmd� kill_proc� panic_node� reboot_node� rg_switchover� rup_nodes

transports.pm � fail_transport� recover_transport

Fault and Event SyntaxYou can arrange the Fault Client Configuration File entries in any order. Lines thatbegin with a # are treated as comments. Comments are ignored. Set the options thatare marked Boolean in Table 7–3 to 0 (False) or 1 (True).

TABLE 7–3 Syntax

Fault and Event Entry Description

dg_switchover device-group node device-group: Device group that you want to switch over.

node: node to switch the device group over to.

fail_pnmd node node: Node on which to fail pnmd.

fail_transport node adapterboolean:autorecover?

node: Node on which to fail the transport.

adapter: Transport adapter that you want to failautorecover?. If true, the transport is automaticallyre-enabled after the failure. Otherwise, the transportremains down.

kill_proc process node process: The process that you want to kill. If you findmultiple processes that match the name that youspecified, the Sun Cluster Fault Server (process)terminates the first process that matches that name thatyou specified.

node: Node on which to kill the process.

panic_node node node: Hostname of the node that you want to panic.

reboot_node node node: Hostname of the node that you want to reboot.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





TABLE 7–3 Syntax (Continued)Fault and Event Entry Description

recover_transport node adapter node: Node on which to recover the transport.

adapter: Transport adapter that you want to recover.

rg_switchover resource-group node resource-group: Resource group that you want to switchover.

node: Node to which you want to switch the resourcegroup.

rup_nodes Entry causes the Sun Cluster Fault Client (process) toinvoke the rup command to request the status of allnodes before and after a fault and event request. TheSun Cluster Fault Client (process) saves this output inthe Fault Client Log.

Fault Client Configuration File KeywordsTo use the faults and events in a Fault Client Configuration File, use the pluginkeyword to request that the Sun Cluster Fault Client (process) register a specificplugin with the Sun Cluster Fault Client (process) and the Sun Cluster Fault Server(process). Because the Fault Client Configuration File is parsed and is processedsequentially, the plugin keyword must appear before any entries that reference faultsfrom that plugin. To avoid a fault or event from being referenced before the pluginloads, place the required plugin entries at the beginning of the Fault ClientConfiguration File.

TABLE 7–4 Keywords

Keyword Description

ignore resgrp|devgrpresource-group

Tells the Sun Cluster Fault Client (process) to ignore the givenresource group or device group from the fault recovery checks. Thefault recovery checks are performed when you use thefailed_node_recovery API call. This ignore keyword is useful ifyou test with a cluster that has a device group or a resource groupthat was intentionally taken offline and put into maintenance state.

resource-group: Name of the resource group you want the Sun ClusterFault Client (process) to ignore.

plugin module module: Name of a Perl module that conforms to the specifications thatare described in “Creating Customized Faults and Events”on page 48.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





TABLE 7–4 Keywords (Continued)Keyword Description

sleep minutes minutes: Number of minutes that you want the Sun Cluster FaultClient (process) to sleep. You can intermix this option between thevarious faults as a throttling mechanism.

Generating a Fault Client ConfigurationFile AutomaticallyThis section contains information on how to generate a Fault Client Configuration File.

The gen_fault_config.pl tool automatically generates a Fault ClientConfiguration File for a cluster. After you install the SCATE packages, this tool residesin /opt/SUNWfltc/bin/gen_fault_config.pl. Table 7–5 lists the command-linearguments that this tool uses.

TABLE 7–5 gen_fault_config.pl Arguments

Option Description

-h hostname hostname: Name of one of the nodes in the cluster that you want to test

This gen_fault_config.pl tool connects to the Sun Cluster FaultServer (process) that runs on this node to query the node for clusterconfiguration information. The gen_fault_config.pl tool uses thisinformation to generate the Fault Client Configuration File. This option isrequired.

-f outputfile outputfile: Preferred path name of the Fault Client Configuration File

By default, the tool generates a faults.conf file in the current workingdirectory.

-p port port: Port on which the Sun Cluster Fault Server (process) listens

The default is 9876.

Generating a Summary of Fault andEvent ResultsThis section contains information on how to generate a summary of the Sun ClusterFault Client (process) log.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





The framework automatically generates a summary of the Sun Cluster Fault Client(process) log when you use the framework to run the Sun Cluster System FaultGenerator.

Both the Sun Cluster Fault Client (process) and the Sun Cluster Fault Server (process)produce time-stamped logs. Sun Cluster Fault Server (process) logs reside in the/var/opt/SUNWflts/ directory on each node. Sun Cluster Fault Client (process)logs save to your current working directory or the value you specify with the --Loption. The default file name for Sun Cluster Fault Client (process) log files is filename faultclient.log.date.

Each time you run the Sun Cluster System Fault Generator, a new Sun Cluster FaultClient (process) log is generated. The Sun Cluster Fault Client (process) log becomeslarge. Use the analyzeFaultLogs.pl tool in the Sun Cluster Fault Client (process)package to analyze the Sun Cluster Fault Client (process) log. After you install thistool, you can locate the tool at /opt/SUNWfltc/bin/analyzeFaultLogs.pl.

TABLE 7–6 analyzeFaultLogs.pl Syntax

Option Description

-llog-file-location This option displays a summary of the Sun Cluster Fault Client(process) log.

log-file-location: Location of the Sun Cluster Fault Client (process) log.

-o This option generates a summary of the Sun Cluster Fault Client(process) log and names the file faultclient.log.date.anal.

Example—Summary of Fault and Event ResultsThis example shows output from the analyzeFaultLogs.pl tool.

Fault Type = Total Injections========== ================Device Group failover 262Resource Group failover 130Reboot Node 130Panic Node 130Power Off Node 5Transport Failure 5Fail pnmd 10phys-schost-1 peak load = 22.15phys-schost-1 low load = 0.03phys-schost-2 peak load = 12.88

phys-schost-2 low load = 0.02


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Creating Customized Faults and EventsThis section contains information on how to use Sun Cluster System Fault Generator’splugin architecture and the available API to implement code to generate new faultsand new events.

You can extend both Sun Cluster Fault Client (process) and Sun Cluster Fault Server(process) capabilities without directly modifying Sun Cluster Fault Client (process)and Sun Cluster Fault Server (process) code. You do not need to learn thecommunication protocol Sun Cluster Fault Client (process) and Sun Cluster FaultServer (process) use.

Sun Cluster Fault Client (process) and Sun Cluster Fault Server (process) implement aplugin model. This plugin model uses Perl modules that contain code extensions.These code extensions load on demand, as Sun Cluster Fault Client (process) parsesthe Fault Client Configuration File. Sun Cluster System Fault Generator providesinterfaces for both Sun Cluster Fault Client (process) and Sun Cluster Fault Server(process). Use these interfaces to build your Perl modules.

Sun Cluster Fault Client (process) InterfaceTable 7–7 lists the functions that you need to use the Sun Cluster Fault Client (process)interface. Table 7–8 lists the data structures that you can use in your code extensionsfor the Sun Cluster Fault Client (process). These functions and data structures exist inSun Cluster Fault Client (process)’s namespace. You can invoke these functions anddata structures as follows.

&main::function_name(args);

TABLE 7–7 Sun Cluster Fault Client (process) Functions

Function Description

check_proc (process,node)

Checks whether a particular process is running on the specified node.

process: The process that you want to check.

node: Node on which to look for process.

RETURNS: 1 if process is running; 0 if process is not running.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





TABLE 7–7 Sun Cluster Fault Client (process) Functions (Continued)Function Description

exec_serverside(wait_for_return, node,function, functionargs)

Provides a mechanism for the client-side implementation of a fault toinitiate the server-side implementation of that fault. Provides amechanism to invoke the Sun Cluster Fault Server (process) interfacecalls from the Sun Cluster Fault Client (process). The Sun ClusterFault Client (process) can use this function to trigger the execution ofa Perl function on a node. This function must reside in the SunCluster Fault Server (process)’s namespace. The plugin architecturesatisfies this configuration requirement. All code extensions reside in asingle Perl module. Both the Sun Cluster Fault Client (process) andthe Sun Cluster Fault Server (process) load this Perl module.

wait_for_return: This value must be a Boolean. If true, the Sun ClusterFault Server (process) waits for the requested function to return beforesending an acknowledgment to the Sun Cluster Fault Client (process).Otherwise, Sun Cluster Fault Server (process) sends theacknowledgment to the Sun Cluster Fault Client (process) beforeexecuting the function. Set this value to false only if the functionthat you want to invoke causes that node to fail. If you request thatthe Sun Cluster Fault Server (process) immediately send anacknowledgment, the Sun Cluster Fault Client (process) does nothang if Sun Cluster Fault Server (process) does not receive anacknowledgment.

node: Node on which to invoke the Perl function.

function: Name of the function to invoke.

functionargs: Comma-delimited list of arguments for the function.

RETURNS: The return code of the function that is being invoked bythe Sun Cluster Fault Server (process).

failed_node_recovery(node, wait_for_node)

Encapsulates all of the code necessary for the Sun Cluster Fault Client(process) to verify a node’s successful recovery from a reboot or panic.This function implements the following tasks.� Verifies that all device groups recovered� Verifies that all resource groups recovered� Verifies that all highly available global file systems recovered

This function causes the Sun Cluster Fault Client (process) to blockadditional faults or cluster events until all of the previous verificationssucceed.

node: The node that failed.

wait_for_node: A boolean that, if true, then this function waits for thenode to reboot before the cluster recovers.

RETURNS: Nothing.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





TABLE 7–7 Sun Cluster Fault Client (process) Functions (Continued)Function Description

fatal_error (error_msg) Causes the Sun Cluster Fault Client (process) to exit an error messagethat is specified by the error_msg argument when Sun Cluster FaultClient (process) code extensions that encounter fatal errors can callthis function.

error_msg: The message to be sent to stdout and the log file.

RETURNS: Nothing.

get_config Updates the data structures when the Sun Cluster Fault Client(process) connects to the Sun Cluster Fault Server (process). The SunCluster Fault Client (process) can call this function to cause these datastructures to be updated. This function updates these data structuresif Sun Cluster Fault Client (process) or Sun Cluster Fault Server(process) code extension results in a change to the clusterconfiguration. The failed_node_recovery function relies onaccurate data in these data structures.

RETURNS: Nothing.

log_message (loglevel,use_timestamp, entry)

Functions as the main interface to the Sun Cluster Fault Client(process)’s logging facility. Messages that are submitted to the loggingfacility are logged to both STDOUT and the fault client log file.

loglevel: Higher numbers indicate a lower priority for the log message.The log level of the Sun Cluster Fault Client (process) is specified onthe command line, indicating the verbosity that you desire.

use_timestamp: A Boolean value that, if true, then this function causesthe log entry to precede a timestamp.

entry: The text string to be logged.

RETURNS: Nothing.

seq_search (element,array)

Performs a simple sequential search of an array.

RETURNS: 1 if the element is found; 0 if the element is not found.

verify_devgrp(devgrp)

Verifies that the device group devgrp exists on the cluster.

RETURNS: 1 if device group exists; 0 if device group does not exist.

verify_node (node) Verifies that the node, node is part of the cluster.

RETURNS: 1 if node is a valid node; 0 if node is not a valid node.

verify_resgrp (resgrp) Verifies that the resource group, resgrp exists on the cluster.

RETURNS: 1 if resource group exists; 0 if resource group does notexist.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





TABLE 7–8 Sun Cluster Fault Client (process) Data Structures

Sun Cluster FaultClient (process) DataStructures

Description

devgrps An array contains all of the device groups which are configured on thecluster

globalFS An array that contains the mount points of all of the global file systemswhich are configured on the cluster

nodes An array of all of the nodes in the cluster

quorumDevs An array of all of the configured quorum devices on the cluster

resgrps An array of all of the configured resource groups on the cluster

Sun Cluster Fault Server (process) InterfaceTable 7–9 lists the functions that you need to use the Sun Cluster Fault Server (process)interface. Table 7–10 lists the data structures that you can use in your code extensionsfor the Sun Cluster Fault Server (process). As stated previously, code extensions to theSun Cluster Fault Server (process) are triggered by a call to the exec_serversideAPI call in the Sun Cluster Fault Client (process)’s code.

TABLE 7–9 Sun Cluster Fault Server (process) Functions

Function Description

checkProcess (process) Checks whether a particular process is running.

process: The process that you want to check

RETURNS: 1 if process is running. 0 if process is notrunning.

getClusterInfo Causes these data structures to be updated with themost current data.

RETURNS: Nothing.

getDGState (devgrp) devgrp: the device group to check

RETURNS: The device group’s status as reported byscstat(1M).

getFSState (fs) fs: file system mount point



Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





TABLE 7–9 Sun Cluster Fault Server (process) Functions (Continued)Function Description

getRGState (rg) rg: Resource group to check

RETURNS: The resource group state as reported byscha_resourcegroup_get(1HA).

log_message(loglevel,use_timestamp, entry)

Functions as the main interface to the Sun Cluster FaultServer (process)’s logging facility. Messages that aresubmitted to the logging facility are logged to bothSTDOUT and the fault server log file.

loglevel: Higher numbers indicate a lower priority forthe log message. The log level of the server is specifiedon the command line, indicating the verbosity that youdesire.

use_timestamp: A Boolean value that if true, causes thelog entry to be preceded by a timestamp.

entry: The text string to be logged.

RETURNS: Nothing.

seqSearch (element, array) Performs a simple sequential search of an array.


TABLE 7–10 Sun Cluster Fault Server (process) Data Structures

Sun Cluster FaultServer (process)data structures

Description

clusterInfo An associative array with the following keys.� devgrps: An array of device groups in the cluster� nodes: An array of nodes in the cluster� resgrps: An array of resource groups in the cluster

globalFS An array of global file systems in the cluster. The Sun Cluster Fault Client(process) and the Sun Cluster Fault Server (process) do not track global filesystems that are not highly available.

quorumDevs An array of the configured quorum devices on the system.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Example—Developing a Fault PluginModuleThe following example demonstrates how to use the Sun Cluster Fault Client (process)interface and the Sun Cluster Fault Server (process) interface to build a fault pluginmodule. This example implements a dg_switchover event. This example includescomments that describe how the Perl module is organized to become a compliant faultplugin.

#!/bin/perl

package example_plugin;

# The Sun Cluster System Fault Generator uses Perl modules to implement fault plugins.# To use any of the faults defined in a plugin, add the following to the# Fault Client Configuration File to instruct the client to register (load) this# Perl module on the client and all nodes:

# plugin <module_name>

# Use this particular module by adding the line "plugin example_plugin" to the# Fault Client Configuration File

# All fault plugins require this array. This array contains the names of entry point# functions that are defined in this module. These entry point functions extend the# capability of the Sun Cluster Fault Client (process). The# Sun Cluster Fault Client (process) can directly call these functions by referencing# these functions in the Fault Client Configuration File. Use the following syntax:

# func_name arg1 arg2 ... arg n

@public_interfaces = ("dg_switchover");

$SCSWITCH = "/usr/cluster/bin/scswitch";

# Think of this implementation as the client-side implementation of the dg_switchover# fault. If the Sun Cluster Fault Server (process) needs to run code on a node,# as is necessary in this case, place that code in a separate function. Invoke that# code by using the exec_serverside API call.

# ARG0 = device group to switchover# ARG1 = node to become primary

# To use the fault, add the following line to the Fault Client Configuration File.

# dg_switchover <device group> <node>

sub dg_switchover{

my $dg = $_[0];


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





my $node = $_[1];

&main::log_message(1, 1, "Switching Device Group $dg over to $node...");

# Skip the verification if you have a raw device group

if ($dg =~ /^dsk\/d\d+/) {}else {

if ( &main::verify_devgrp($dg) != 0 ) {&main::log_message(1, 0, "Skipping - $dg is not a valid device group");return (-1);

}}if ( &main::verify_node($node) != 0 ) {

&main::log_message(1, 0, "Skipping - $node is not a valid cluster node");return (-1);

}if ( &main::exec_serverside(1, $node, "standard::dg_switchover_server", "$dg,$node") != 0)&main::log_message(1, 1, "Device Group switchover failed!");&main::log_message(1, 0, "Please see the fault server logs on $node for more details.");

}

return (0);}

# The fault server daemon executes this function on a node. The "dg_switchover" function# defined above, through an exec_serverside API call, triggers this function.

# ARG0 = device group to switchover# ARG1 = physical node to takeover device group

sub dg_switchover_server{

my $dg = $_[0];my $host = $_[1];

&main::log_message(1, 1, "Executing command: $SCSWITCH -z -D $dg -h $host");my $exitcode = system("$SCSWITCH -z -D $dg -h $host");

if ($exitcode == 0) {&main::log_message(1, 0, " ret code = $exitcode");return (0);

} else {$exitcode = $exitcode / 256;&main::log_message(1, 0, " ret code = $exitcode");return ($exitcode);

}}

;


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





APPENDIX A

Understanding the Sun ClusterAutomated Test Environment SoftwareFramework

This appendix provides conceptual information about the Sun Cluster Automated TestEnvironment software framework.

Sun Cluster Automated TestEnvironment Software FrameworkFeaturesThis section contains a list of features in Sun Cluster Automated Test Environmentsoftware (SCATE).

The SCATE is a suite of programs that work together to provide automated testexecution. The SCATE includes the following features.

� Sun Cluster Automated Tests

These tests consist of functional tests, load tests, and load tests with faults. The SunCluster Functional Tests verify that the hardware you intend to qualify meets theSun Cluster software’s expectations for communication with devices. The SunCluster Load Tests and the Sun Cluster Load Tests With Faults verify that thehardware you intend to qualify and the Sun Cluster software operate as expectedunder load and fault injection.

� Sun Cluster System Fault Generator

These features provide the tools to verify that the following software and hardwarecomponents are compatible with Sun Cluster software.

� Private interconnect� Public network� Server platform

55

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





� Storage

Sun Cluster Automated TestEnvironment Software FrameworkComponentsThis section contains descriptions of the components that comprise the Sun ClusterAutomated Test Environment software (SCATE) framework.

Figure A–1 shows how these components work together. Figure A–1 also shows whichcomponents you need to test to qualify your private interconnect with Sun Clustersoftware.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





PublicNetwork

scaterun

Sun ClusterFault Client(process)

SCATEServer

(process)

SCATE Server (system)

TestRequest

File

Test Results File

Client 1

Client

MultihostDisks

Sun ExplorerData Collector

Sun Cluster FaultServer (process)

Node 1



Node 2

Public network interface



Node N

Clustertransportadapters

Clustertransportcables

NAFOgroup

DataService

DataService

DataService


NAFOgroup


NAFOgroup

Components that require functional, load and fault testing

FIGURE A–1 SCATE Framework

Appendix A • Understanding the Sun Cluster Automated Test Environment Software Framework 57

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





The following components comprise the SCATE framework.

� “scaterun Utility” on page 58� “SCATE Server (system)” on page 59� “SCATE Server (process)” on page 58� “Sun Explorer Data Collector” on page 58� “Sun Cluster System Fault Generator” on page 58� “Sun Cluster Fault Client (process)” on page 59� “Sun Cluster Fault Server (process)” on page 59

scaterun UtilityThe scaterun utility is the SCATE interface and interactive script that enables you torun the Sun Cluster Functional Tests, the Sun Cluster Load Tests, and the Sun ClusterLoad Tests With Faults from one location. After you select the tests you intend to run,the scaterun utility prompts you for answers to specific questions and generates aTest Request File. The scaterun utility uses this Test Request File to communicatewith the “SCATE Server (process)” on page 58.

SCATE Server (process)The SCATE Server (process) is a daemon that runs on the SCATE Server (system). TheSCATE Server (process) initiates, schedules, executes, monitors, and managesautomated tests. The SCATE Server (process) starts and stops the Sun Cluster FaultClient (process). Lastly, when the scaterun utility sends a Test Request File to theSCATE Server (process), the SCATE Server (process) provides the scaterun utility aTest Results File. The SCATE names the Test Results File SCATE_REPORT.

Sun Explorer Data CollectorThe Sun Explorer Data Collector is a program that captures a snapshot of each node’sconfiguration information. The program’s output provides information you need totroubleshoot test problems. The Sun Explorer Data Collector captures its snapshotbefore you run the automated tests.

Sun Cluster System Fault GeneratorThe Sun Cluster System Fault Generator is a client-server application that enablesusers to simulate system faults and events that run on a cluster. The Sun ClusterSystem Fault Generator includes two processes. These two processes are the “SunCluster Fault Client (process)” on page 59 and the “Sun Cluster Fault Server(process)” on page 59.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Sun Cluster Fault Client (process)The Sun Cluster Fault Client (process) is a client that sends faults or event requests tothe “Sun Cluster Fault Server (process)” on page 59 and queries the nodes to verifyproper recovery from fault or event request. The Sun Cluster Fault Client (process)runs on the SCATE Server (system).

Sun Cluster Fault Server (process)The Sun Cluster Fault Server (process) is a daemon that listens for requests from the“Sun Cluster Fault Client (process)” on page 59 and generates fault or event requestson the cluster. The Sun Cluster Fault Server (process) runs on all nodes.

SCATE Server (system)The SCATE Server (system) is a system or machine that resides outside of the cluster.

Appendix A • Understanding the Sun Cluster Automated Test Environment Software Framework 59

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





APPENDIX B

Understanding the Sun ClusterAutomated Tests

This appendix provides conceptual information on the Sun Cluster Automated Tests.

The Sun Cluster Automated Tests that you use to qualify your hardware on the SunCluster software can be grouped in the following categories. This section describeseach test within these categories. This section also describes how to use the scaterunutility to run the Sun Cluster Automated Tests.

� “Sun Cluster Functional Tests” on page 61� “Sun Cluster Load Tests” on page 64� “Sun Cluster Load Tests With Faults” on page 65

Sun Cluster Functional TestsThis section contains conceptual information on Sun Cluster Functional Tests.

The Sun Cluster Functional Tests verify that the hardware that you intend to qualifycorrectly handles low-level operations. The Sun Cluster Functional Tests also verifythat the hardware that you intend to qualify meets the Sun Cluster software’sexpectations for communicating with devices. This section provides descriptions forthe following Sun Cluster Functional Tests.

� “Cluster File System Functional Tests” on page 62� “Cluster Interconnect Functional Tests” on page 62� “Global Device Functional Tests” on page 62� “NAFO Functional Tests” on page 62� “Oracle Parallel Server/Real Application Clusters Functional Tests” on page 63� “RGM Functional Tests” on page 63� “Scalable Service Functional Tests” on page 63� “Solstice DiskSuite Functional Tests” on page 63� “VERITAS Volume Manager Functional Tests” on page 63

61

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Cluster File System Functional TestsThis test suite verifies the following cluster file system capabilities.

� Global mount capability� Location-transparent access to files� Caching and coherency protocols that involve concurrent file access from multiple

nodes� File systems, directories, and file operations on a cluster file system� Continuous data access under node failures and switchovers, assuming the path to

the disk is operational

Cluster Interconnect Functional TestsThis test suite verifies the following cluster interconnect capabilities.

� Ability to add, remove, and change the properties of a cluster transport adapter� Ability to add, remove, and change the properties of a cluster transport junction� Ability to add, remove, and change the properties of a cluster transport cable� Highly available communication of Sun Cluster infrastructure components and

user applications between cluster nodes

Recovery from a failure of a cluster transport adapter, a cluster transport junction,or a cluster transport adapter

Global Device Functional TestsThis test suite verifies the following global device capabilities.

� Under no node failures, access to multihost disks from all nodes� Under node failures, access to multihost disks from all nodes

NAFO Functional TestsThis test suite verifies the following Network Adapter Failover (NAFO) capabilities.

� Ability to monitor public network adapters� Failover of shared and failover IP addresses from one adapter to another� Availability of high-level connections such as TCP during a failover of an IP

address


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Oracle Parallel Server/Real Application ClustersFunctional TestsThis test suite verifies the following OPS/RAC capabilities.

� Creates a sample database and starts Oracle instances on each node� Verifies that both local and remote clients can query and update the database� Verifies that the consistency of the database is maintained across all nodes when

updates are performed in parallel with software fault injections� Verifies that the consistency of the database is maintained across all nodes during

updates

RGM Functional TestsThis test suite verifies the following Resource Group Manager (RGM) capabilities.

� Ability of RGM to create and manage resources within resource groups� Ability to start and stop resource groups on selected nodes in response to various

events

Scalable Service Functional TestsThis test suite verifies the following scalable services framework capabilities.

� Networking infrastructure support for scalable services� Ability to create scalable TCP and UDP services with instances that run on multiple

nodes and that behave as a single service� Load distribution of client requests for weighted load-balancing policy

Solstice DiskSuite Functional TestsThis test suite verifies the following Solstice DiskSuite/Solaris Volume Managercapabilities.

� Diskset creation by using disk device groups� Disk device group access by using global devices� Global device access by using a cluster file system or raw devices� Failure fencing

VERITAS Volume Manager Functional TestsThis test suite verifies the following VERITAS Volume Manager (VxVM) capabilities.

� Disk group creation using disk device groups

Appendix B • Understanding the Sun Cluster Automated Tests 63

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





� Disk device group access using global devices� Global device access, using raw devices or UFS cluster file systems, after a global

mount� Continuous data access, especially under node failures and switchovers

Sun Cluster Load TestsThis section contains conceptual information on Sun Cluster Load Tests.

The Sun Cluster Load Tests verify that the hardware that you intend to qualify and thecluster operate as expected under load. Each load test has a Load Generator. LoadGenerators are tools that generate load on one or more file systems, on raw devices,and on public or private networks. This section provides descriptions for the followingSun Cluster Load Tests.

� “Sun Cluster HA for NFS Load Tests” on page 64� “Global Device Load Tests” on page 64� “Scalable Service Load Tests” on page 65� “Sun Cluster HA for Apache/Sun Cluster HA for Sun ONE Web Server Load

Tests” on page 65� “Sun Cluster HA for Oracle Load Tests” on page 65

Sun Cluster HA for NFS Load TestsUse this test suite to test a cluster file system and Sun Cluster HA for NFS. This testsuite consists of the following tests.

� I/O Asynchronous Load Test

This test spawns multiple processes with multiple threads for each process. Eachthread opens a file and performs random, asynchronous I/O operations on the file.

� I/O Synchronous Load Test

This test spawns multiple processes with multiple threads for each process. Eachthread opens a file. Each thread performs random, synchronous I/O operations onthis file.

Global Device Load TestsThis test suite generates a large number of synchronous and a large number ofasynchronous read and write requests to a set of global devices from all nodes. Thissuite also verifies that the cluster is operational and servicing I/O requests.


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Scalable Service Load TestsThis test suite generates load on the cluster by spawning a number of processes on aset of client machines. These processes create a combination of short-lived connectionsand persistent connections to specific TCP-based scalable services. These specificTCP-based scalable services include TCP-based scalable services that run on all nodesand that exchange data over the TCP connections. This test suite’s function is to testthe scalable services infrastructure and load distribution of the TCP connectionsamong all the cluster nodes under load.

Sun Cluster HA for Apache/Sun Cluster HA forSun ONE Web Server Load TestsThis test suite works with both Apache Web Server and Sun ONE Web Server. The testsuite configures Apache Web Server and Sun ONE Web Server as a scalable dataservice on all nodes. The test suite creates load by creating a number of virtual userson a user-defined set of client machines. These client machines simultaneouslygenerate HTTP GET requests to the scalable data service.

Sun Cluster HA for Oracle Load TestsUse this test suite to test Sun Cluster HA for Oracle on a cluster file system or rawdevices. This test suite consists of the following components.

� A Test Database on a global file system or on raw devices.

� A load test that spawns a variable number of Oracle client processes. Each processperforms a random mix of queries and updates to the Test Database.

� A utility to parse the load test logs to generate a report with transaction statistics.

Sun Cluster Load Tests With FaultsThis section contains conceptual information on Sun Cluster Load Tests With Faults.

The Sun Cluster Load Tests With Faults consist of the Sun Cluster Load Tests withfault injections. The SCATE uses the “Sun Cluster System Fault Generator” on page 58to generate faults. This utility runs in parallel with the Sun Cluster Load Tests, creatingthe Sun Cluster Load Tests With Faults.

Appendix B • Understanding the Sun Cluster Automated Tests 65

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Glossary

qualify The act of testing software or hardware to ensure that it operates asexpected within a specific configuration.

scatesetup The scatesetup utility is an interactive script that can be used toinstall the appropriate set of SCATE packages on all nodes, clientmachines, and the SCATE Server (system). The scatesetup utilitysupports a full SCATE installation on all machines, on a new machinebeing added to an existing environment, and a standalone Sun ClusterFault Server (process) installation.

test case The units that constitute a test suite.

test execution The act of performing all steps in a test scenario, as outlined by a testplan, to observe and collect data that supports the test results. Thisdata is compared against the expected test results to determine if thetest scenario passed or failed.

test purpose The units that constitute a test case.

test purpose number The steps within a test purpose. Each test purpose is associated with atest purpose number.

67

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Index

Aaborts, 36analyzeFaultLogs.pl tool, 47analyzing, test results, 39API, 48

Cclients, requirements, 20cluster file system

description, 64requirements, 24

Cluster File System Functional Testsdescription, 62requirements, 22

Cluster Interconnect Functional Testsdescription, 62requirements, 23

Ddata structures

Sun Cluster Fault Client (process)interface, 51

Sun Cluster Fault Server (process)interface, 52

dg_switchover, 44

Eenvironment

framework, 56launching, 30

events, 44customizing, 48implementing, 43

exec file, 38Explorer Data Collector, 58

Ffail_pnmd, 44fail_transport, 44Fault Client Configuration File

entries, 44generating, 46

Fault Generator, description, 58Fault Server (process), description, 59faultclient.pl tool, 43faults, 44

customizing, 48implementing, 43

functional testsCluster File System Functional Tests, 22, 62Cluster Interconnect Functional Tests, 23, 62description, 61Global Device Functional Tests, 23, 62NAFO Functional Tests, 62OPS/RAC Functional Tests, 23, 63requirements, 22RGM Functional Tests, 63

69

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





functional tests (Continued)Scalable Service Functional Tests, 23, 63Solstice DiskSuite Functional Tests, 63VERITAS Volume Manager Functional

Tests, 63functions

Sun Cluster Fault Client (process)interface, 48

Sun Cluster Fault Server (process)interface, 51

Ggen_fault_config.pl tool, 46generating, Test Results File, 38Global Device Functional Tests


Global Device Load Testsdescription, 64requirements, 24

Hhardware, 19help, 10

Iinstalling

packages, 15Sun Cluster Automated Test Environment

software, 15

Kkill_proc, 44

Llaunching, test environment, 30load generators, 24

load testscluster file system, 24, 64description, 64Global Device Load Tests, 24, 64Scalable Service Load Tests, 24, 65Sun Cluster HA for Apache/Sun Cluster HA

for Sun ONE Web Server Load Tests, 24,65

Sun Cluster HA for NFS Load Tests, 24, 64Sun Cluster HA for Oracle Load Tests, 65

load tests with faults, Sun Cluster Load TestsWith Faults, 65

NNAFO Functional Tests, description, 62nodes, requirements, 20

OOPS/RAC Functional Tests


Ppackage, SUNWflts, 17packages

installing, 15Perl Database Interface, 17Perl for Solaris, 17Sun Cluster Automated Test Environment

software, 17Sun Cluster Fault Client (process), 17Sun Cluster Fault Server (process), 17Sun Explorer Data Collector, 17SUNWexplo, 17SUNWfltc, 17SUNWflts, 17SUNWpldbi, 17SUNWscts, 17

panic_node, 44parameters, 35Perl Database Interface, packages, 17


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





pluginslist of, 44registering, 45

processes, 36

Qqualification process, 11

Rreboot_node, 44recover_transport, 44, 45requirements

clients, 20cluster file system, 24Cluster File System Functional Tests, 22Cluster Interconnect Functional Tests, 23Global Device Functional Tests, 23Global Device Load Tests, 24hardware, 19nodes, 20OPS/RAC Functional Tests, 23Scalable Service Functional Tests, 23Scalable Service Load Tests, 24SCATE Server (system), 20Sun Cluster Automated Tests, 21Sun Cluster HA for Apache/Sun Cluster HA

for Sun ONE Web Server Load Tests, 24Sun Cluster HA for NFS Load Tests, 24

retrieving, test results, 37rg_switchover, 44, 45RGM Functional Tests, description, 63running, tests, 31rup_nodes, 44, 45

SScalable Service Functional Tests


Scalable Service Load Testsdescription, 65requirements, 24

SCATE_REPORT file, 37

SCATE Server (process)description, 58starting, 29

SCATE Server (system)description, 59requirements, 20

scaterun utility, 30, 58setting up, Sun Cluster Automated Test

Environment software, 13Solstice DiskSuite Functional Tests,

description, 63standard.pm, 44starting, SCATE Server (process), 29status, Sun Cluster Automated Tests, 36summary file, 37Sun Cluster Automated Test Environment

software, 15packages, 17setting up, 13using, 29

Sun Cluster Automated Tests, test status, 36Sun Cluster Fault Client (process)

description, 59options, 42packages, 17

Sun Cluster Fault Server (process),packages, 17

Sun Cluster HA for Apache/Sun Cluster HA forSun ONE Web Server Load Testsdescription, 65requirements, 24

Sun Cluster HA for NFS Load Testsdescription, 64requirements, 24tuning, 32

Sun Cluster HA for Oracle Load Tests,description, 65

Sun Cluster Load Tests, optimal load testparameters, 35

Sun Explorer Data Collector, packages, 17SUNWexplo package, 17SUNWfltc package, 17SUNWflts package, 17SUNWpl5u package, 17SUNWpldi package, 17SUNWscts package, 17

Index 71

Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





Ttechnical support, 10Test Details File, 37test log directory, 37Test Request File, 59Test Results File, 37, 38Test Summary File, 37tests

See also environmentrecovering, 36requirements, 21running, 31test results, 37, 39

thresholds, 35transports.pm, 44tuning, Sun Cluster HA for NFS Load Tests, 32

Uusing, Sun Cluster Automated Test

Environment software, 29

V/var/opt/SUNWflts directory, 47VERITAS Volume Manager Functional Tests,

description, 63


Click t


ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.comClic

k to BUY NOW!PD

F-XChange Editor

ww


om Click t

o BUY NOW!PD

F-XChange Editor

ww

w.tracker-software

.com





sun cluster automated test environment 1.2 private ...sun cluster automated test environment 1.2...

Documents