peoplesoft v8 on zseries using sysplex data sharing haringthis information could include technical...

ibm.com/redbooks

PeopleSoft V8 on zSeries Using Sysplex Data Sharing haring and Enterprise Storage Systems

Viviane Anavi-ChaputTroy Callanan

Fred OroscoDave Perrault

Dipendra WaykarBill Worthington

zSeries sysplex data sharing and PeopleSoft

DB2 Connect and load balancing

ESS performance with PeopleSoft

Front cover

http://www.redbooks.ibm.com/http://www.redbooks.ibm.com/

PeopleSoft V8 on zSeries Using Sysplex Data Sharing and Enterprise Storage Systems

March 2004

International Technical Support Organization

SG24-6093-00

© Copyright International Business Machines Corporation 2004. All rights reserved.Note to U.S. Government Users Restricted Rights -- Use, duplication or disclosure restricted by GSA ADPSchedule Contract with IBM Corp.

First Edition (March 2004)

This edition applies to PeopleSoft Version 8.4, DB2 for z/OS Version 7, and DB2 Connect EE Version 8.

Note: Before using this information and the product it supports, read the information in “Notices” on page vii.

Contents

Notices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiTrademarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viii

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixThe team that wrote this redbook. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixBecome a published author . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xComments welcome. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . x

Chapter 1. Benefits of zSeries sysplex data sharing for PeopleSoft . . . . . 11.1 Why Parallel Sysplex and data sharing for PeopleSoft . . . . . . . . . . . . . . . . 2

1.1.1 Continuous availability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.1.2 Processor scalability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 Parallel Sysplex architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 DB2 data sharing architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.3.1 Data sharing configurations for high availability and scalability . . . . . 61.3.2 How many data sharing groups? . . . . . . . . . . . . . . . . . . . . . . . . . . . 111.3.3 How many data sharing members? . . . . . . . . . . . . . . . . . . . . . . . . . 13

Chapter 2. Performance considerations: implementing PeopleSoft with Parallel Sysplex data sharing . . . . . . . . . . . . . . . . . . . . . . . . . . 17

2.1 Sysplex performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.1.1 CF performance monitoring and tuning. . . . . . . . . . . . . . . . . . . . . . . 182.1.2 WLM policies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222.1.3 RMF Monitor III - Work Delay Monitor. . . . . . . . . . . . . . . . . . . . . . . . 252.1.4 RMF post processor reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272.1.5 RMF Spreadsheet Reporter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

2.2 Data sharing performance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302.2.1 Local buffer pool performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 312.2.2 Group buffer pool performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322.2.3 Dynamic statement cache. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362.2.4 Lock performance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 372.2.5 DB2PM statistics batch report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432.2.6 Table space partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 442.2.7 Managing DBM1 virtual storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . 472.2.8 Configuring DB2 sort pool and workfiles . . . . . . . . . . . . . . . . . . . . . . 50

Chapter 3. DB2 Connect architectures for PeopleSoft . . . . . . . . . . . . . . . 533.1 PeopleSoft connectivity architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 543.2 DB2 Connect EE configurations with PeopleSoft . . . . . . . . . . . . . . . . . . . 55

© Copyright IBM Corp. 2004. All rights reserved. iii

3.2.1 Direct connection configuration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553.2.2 Connection server configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

3.3 DB2 Connect EE thread pooling techniques . . . . . . . . . . . . . . . . . . . . . . . 573.3.1 Connection pooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 573.3.2 Connection concentrator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 603.3.3 Connection pooling versus connection concentrator . . . . . . . . . . . . 633.3.4 Sysplex-aware connections . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

3.4 Networking considerations: VIPA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Chapter 4. DB2 Connect EE setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674.1 PeopleSoft test environment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684.2 DB2 Connect EE configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69

4.2.1 Using CCA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 704.2.2 Using CLP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78

4.3 Application server domain configuration . . . . . . . . . . . . . . . . . . . . . . . . . . 794.4 Validating Sysplex awareness setup in DB2 Connect. . . . . . . . . . . . . . . . 79

4.4.1 DRDA trace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 794.4.2 Test program to check load balancing . . . . . . . . . . . . . . . . . . . . . . . 81

Chapter 5. Functional test of load balancing . . . . . . . . . . . . . . . . . . . . . . . 835.1 Application workload used for the test. . . . . . . . . . . . . . . . . . . . . . . . . . . . 845.2 Monitoring load balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 845.3 Monitoring failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

Chapter 6. Volume tests of load balancing . . . . . . . . . . . . . . . . . . . . . . . . . 896.1 Load balancing: CPU saturation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 906.2 Load balancing: additional member . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91

Chapter 7. ESS performance considerations for PeopleSoft . . . . . . . . . . 957.1 Introducing ESS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96

7.1.1 The IBM TotalStorage Enterprise Storage Server Model 800. . . . . . 967.2 Parallel Access Volume. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98

7.2.1 PAV Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 987.2.2 Dynamic and static PAVs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

7.3 FlashCopy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1047.3.1 Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1047.3.2 Database cloning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105

7.4 I/O priority queuing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1087.5 FICON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

Appendix A. PeopleSoft data sharing configuration (option 1): customer example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111

Customer configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112Why data sharing? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

iv PeopleSoft V8 on zSeries Using Sysplex Data Sharing and Enterprise Storage Systems

Designing PeopleSoft in a data sharing environment . . . . . . . . . . . . . . . . . . 113Partitioning schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114Data sharing issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

Appendix B. Test program to validate load balancing . . . . . . . . . . . . . . . 117

Appendix C. FlashCopy using Mainstar MS/VCR. . . . . . . . . . . . . . . . . . . 125

Related publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129Other publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129Online resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129How to get IBM Redbooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129Help from IBM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

Contents v

vi PeopleSoft V8 on zSeries Using Sysplex Data Sharing and Enterprise Storage Systems

Notices

This information was developed for products and services offered in the U.S.A.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.

COPYRIGHT LICENSE: This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces.

© Copyright IBM Corp. 2004. All rights reserved. vii

TrademarksThe following terms are trademarks of the International Business Machines Corporation in the United States, other countries, or both:

AIX®DB2®DB2 Connect™DRDA®Enterprise Storage Server®ESCON®FICON™FlashCopy®Hiperspace™

IBM®Informix®MVS™OS/390®OS/400®Parallel Sysplex®PR/SM™Redbooks (logo) ™RMF™

S/390®Seascape®TotalStorage®WebSphere®z/Architecture™z/OS®zSeries®

The following terms are trademarks of other companies:

Intel, Intel Inside (logos), MMX, and Pentium are trademarks of Intel Corporation in the United States, other countries, or both.

Microsoft, Windows, Windows NT, and the Windows logo are trademarks of Microsoft Corporation in the United States, other countries, or both.

Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

UNIX is a registered trademark of The Open Group in the United States and other countries.

SET, SET Secure Electronic Transaction, and the SET Logo are trademarks owned by SET Secure Electronic Transaction LLC.

Other company, product, and service names may be trademarks or service marks of others.

viii PeopleSoft V8 on zSeries Using Sysplex Data Sharing and Enterprise Storage Systems

Preface

This IBM® Redbook describes the implementation of PeopleSoft 8.4 on zSeries® Parallel Sysplex® data sharing environment. The book discusses the benefits of running PeopleSoft in a sysplex with data sharing. It explains the performance considerations and issues when implementing PeopleSoft in a data sharing environment.

The book also investigates the load balancing of PeopleSoft workloads using DB2® Connect to connect to the sysplex DB2 data sharing members. It describes functional and volume tests of load balancing. A section about Enterprise Storage Systems describes the PeopleSoft performance benefits when using ESS.

The team that wrote this redbookThis redbook was produced by a team of specialists from around the world working at the International Technical Support Organization, Poughkeepsie Center.

Viviane Anavi-Chaput is a Senior IT Specialist for DB2, BI, and ERP at the IBM International Technical Support Organization, New York. She writes extensively, teaches worldwide, and presents at international conferences on all areas of Business Intelligence, ERP, and DB2 for OS/390®. Before joining the ITSO in 1999, Viviane was a Senior Data Management Consultant at IBM Europe, France. She was also an ITSO Specialist for DB2 at the San Jose Center from 1990 to 1994.

Troy Callanan is a DB2 Database Administrator at 3M. He has worked with DB2 since 1992 and with PeopleSoft since 1998. Troy holds a B.S. degree in Business Computer Information Systems from St. Cloud State University.

Fred Orosco is an IT consultant at the IBM Silicon Valley Laboratory - Integration Center for DB2, specializing in PeopleSoft and Siebel integrations on zSeries solutions.

Dave Perrault is an I/T Specialist in the IBM Advanced Technical Support organization, providing technical support to PeopleSoft customers with implementations on DB2 for the IBM zSeries platform. Dave has more than 15 years of experience in large systems and has been with IBM since 1996. With IBM, he has provided technical support for Business Partners porting

© Copyright IBM Corp. 2004. All rights reserved. ix

applications to IBM DB2 database product offerings, including porting assessments and feasibility studies, benchmarking/performance analysis, and pre/post-sales customer technical support.

Dipendra Waykar is an Advisory Software Engineer in PeopleSoft Integration Center at IBM, Silicon Valley Lab. He has more than 11 years of experience in database-server and client-server technologies. Before joining PeopleSoft Integration Center in late 2002, Dipendra was a lead engineer at IBM Informix® Data Management Division and has extensively worked on development and testing of database servers and client APIs.

Bill Worthington is an IBM Consulting IT Specialist in the United States. He has 43 years of experience in data processing, including 39 years in technical sales support within IBM. His areas of expertise include disk storage products. In his current position, Bill is supporting storage products used by independent software vendors including PeopleSoft.

Thanks to the following people for their contributions to this project:

Cleve Bench, John Grey, Randall Hicks, Mark Hoernemann, Jon Nolting, PeopleSoft Inc., Pleasanton, CA, US

Tom Kennelly, Michael CurtisIBM ERP Competency Center, US

Become a published authorJoin us for a two- to six-week residency program. Help write an IBM Redbook dealing with specific products or solutions while getting hands-on experience with leading-edge technologies. You will team with IBM technical professionals, Business Partners, and/or customers.

Your efforts will help increase product acceptance and customer satisfaction. As a bonus, you will develop a network of contacts in IBM development labs, and increase your productivity and marketability.

Find out more about the residency program, browse the residency index, and apply online at:

ibm.com/redbooks/residencies.html

Comments welcomeYour comments are important to us.

x PeopleSoft V8 on zSeries Using Sysplex Data Sharing and Enterprise Storage Systems

http://www.redbooks.ibm.com/residencies.htmlhttp://www.redbooks.ibm.com/residencies.html

We want our Redbooks to be as helpful as possible. Send us your comments about this or other Redbooks in one of the following ways:

� Use the online Contact us review redbook form found at:

ibm.com/redbooks

� Send your comments in an e-mail to:

[email protected]

� Mail your comments to:

IBM Corporation, International Technical Support OrganizationDept. HYJ Mail Station P0992455 South RoadPoughkeepsie, NY 12601-5400

Preface xi

http://www.redbooks.ibm.com/http://www.ibm.com/redbooks/http://www.ibm.com/redbooks/http://www.redbooks.ibm.com/contacts.html

xii PeopleSoft V8 on zSeries Using Sysplex Data Sharing and Enterprise Storage Systems

Chapter 1. Benefits of zSeries sysplex data sharing for PeopleSoft

This chapter introduces the sysplex data sharing benefits for PeopleSoft. We discuss the following topics:

� Why Parallel Sysplex and data sharing for PeopleSoft?

– Continuous availability– Processor scalability

� Parallel Sysplex architecture

� Data sharing architectures

– Data sharing configurations for high availability and scalability

1

© Copyright IBM Corp. 2004. All rights reserved. 1

1.1 Why Parallel Sysplex and data sharing for PeopleSoftThere are several motivations for pursuing a PeopleSoft implementation based on zSeries Parallel Sysplex and DB2 data sharing. Many customers are deploying PeopleSoft applications in support of business-critical operations. Typical business drivers include a desire to run a single global PeopleSoft instance, customer and supplier access to Web-based PeopleSoft applications around the clock, and support of 24x7 manufacturing and distribution operations. Those business drivers lead to the following IT requirements:

� Near-continuous system availability� Central processor scalability through horizontal growth

1.1.1 Continuous availabilityThese business applications require what is known as near-continuous availability. Availability in this context means the ability of all or some of the end users and applications to access the production databases. The question of what percentage of work can be processed on a continuous basis is an economic decision based on cost and the value of continuous availability to the organization. Although there is no standard definition of near-continuous availability, many large companies have adopted a goal of no more than five to ten hours per year of planned plus unplanned outages. This translates to an overall system availability of 99.9%. This concept of availability contrasts to the historical paradigm of a maintenance window of 8 hours every weekend, or 400 hours of outage per year. Other companies are adopting service level objectives slightly less demanding but still challenging. For example, an objective of a quarterly four-hour maintenance window (that is, 16 hours per year of planned outage) requires many of the same design principles as the near-continuous availability objective.

Continuous availability is a combination of high availability and continuous operations.

High availabilityThe first component of continuous availability is high availability. High availability is the ability to avoid unplanned outages. This is typically achieved through reliable quality components, internal redundancy, and sophisticated recovery or correction mechanisms. This applies to software as well as hardware. z/OS has millions of lines of code devoted to correction and recovery. This reliability coupled with a design that eliminates single points of failure at the component level provides what is known in the industry as high availability. In fact, this is high reliability, the ability to avoid unplanned incidents. To achieve continuous

2 PeopleSoft V8 on zSeries Using Sysplex Data Sharing and Enterprise Storage Systems

availability we strive for a design that has 99.999% reliability, which equates to less than five minutes of unplanned downtime per year.

Continuous operationsThe other component of continuous availability is continuous operations. Continuous operations is the ability to avoid planned outages. This includes the ability to upgrade and maintain the central processors, the operating systems, DB2, and coupling facilities without taking the PeopleSoft database down. It also means that DB2 database maintenance must be done while applications are reading and writing data. This requires Online Reorganization, Online Image Copy, and the ability to non-disruptively flash or snap volumes of data.

1.1.2 Processor scalabilityHistorically systems grew vertically by adding processors to the machine (or CEC) or by introducing faster processors. This approach limited the size of a system to the largest single CEC. Systems were also constrained by the amount of data and control information that could be held in the primary DB2 address space DBM1. zSeries Parallel Sysplex architecture enables us to overcome these constraints by clustering multiple CECs in a single operating system image including DB2 data sharing. This enables scalability through horizontal growth of both processor power (MIPs) and virtual storage. Parallel Sysplex also gives us workload management capabilities, as we can now level multiple workloads across two or more machines.

1.2 Parallel Sysplex architectureA fundamental capability to support both high availability and processor scalability is the technology of clustered database servers operating against a single copy of the data.

zSeries Parallel Sysplex technology uses a form of clustering where all servers appear to the business application as a single server. This form of clustering, known as single system image, enables Parallel Sysplex to provide industry-leading availability by enabling workloads to be balanced across multiple servers. Up to 32 servers can be linked together with near-linear scalability, building on and extending the strengths of the zSeries hardware architecture. Every server in a zSeries Parallel Sysplex cluster has access to all data resources, and every cloned application can run on every server. Using zSeries coupling technology, Parallel Sysplex technology provides a shared data clustering technique that permits multi-system data sharing with high-performance read/write integrity. This shared data (as opposed to “shared

Chapter 1. Benefits of zSeries sysplex data sharing for PeopleSoft 3

nothing”) approach enables workloads to be dynamically balanced across all servers in the Parallel Sysplex cluster.

PeopleSoft with its database on DB2 for z/OS® can fully utilize zSeries Parallel Sysplex, which enables PeopleSoft to take advantage of the aggregate capacity of multiple servers for what is probably the most critical part of a PeopleSoft installation, the database server. zSeries Parallel Sysplex helps to ensure maximum system throughput and performance during peak processing periods. In the event of a hardware or software outage, either planned or unplanned, workloads can be dynamically redirected to available servers, providing continuous application availability.

Figure 1-1 introduces a high-level picture of the elements of a zSeries Parallel Sysplex shared data architecture.

Figure 1-1 zSeries Parallel Sysplex shared data architecture elements

In Figure 1-1 we see that a Parallel Sysplex system infrastructure is typically made up of two or more computer systems known as a Central Electronic Complex (CEC), two or more special purpose zSeries computers known as Coupling Facilities (either internal, ICF, or external) for the storing of shared Operating System and DB2 structures between the CECs, two external time sources called sysplex timers, sysplex-wide shared data, and multiple

Shared Data

Sysplex Timer

12 1234

567891011

XCF

XCF CouplingFacility

12 1234

567891011

9672/z900/z990 9672/z900/z990


high-speed, duplexed links connecting the components. This implementation employs hardware, software, and microcode.

As zSeries Parallel Sysplex system is designed for continuous operation, you should make sure that your configuration follows the principle of avoiding single points of failures (SPOFs) to really achieve your availability goal. The key design point is that the configuration should tolerate a failure in any single major component. The design concept of redundancy also applies to zSeries Parallel Sysplex. If at least two instances of a resource exist, then a failure of one allows the application to continue. Sysplex timer, coupling facility, and coupling facility links are elements that should be duplicated.

1.3 DB2 data sharing architecturesIn Figure 1-2 we complete the picture by laying multiple DB2 data sharing members on top of the zSeries Parallel Sysplex infrastructure.

Figure 1-2 DB2 data sharing in a Parallel Sysplex

In this picture we see up to 32 DB2 subsystems (also called DB2 members) making up a DB2 data sharing group. There will be one DB2 data sharing group for each production PeopleSoft system. Each DB2 subsystem has its own set of DB2 logs, local buffer pools, and local locks managed by a companion IRLM. The DB2 data sharing group shares the DB2 databases, the DB2 catalog/directory,

Coupling FacilitySysplex Timer

...

12 1234

567891011

Shared Disks

Data Sharing Group

GlobalLocks

GroupBuffer Pools

Locks

DB2ALocks Buffer Pools

DB2A

SharedSharedDB2 DataDB2 Data

SharedSharedDB2 CatalogDB2 Catalog

DB2 Subsystems

Locks Buffer Pools

DB2A

12 1234

567891011

Buffer Pools

DB2A DB2A LogLog

DB2B DB2B LogLog

DB2n DB2n LogLog


and the DB2 data sharing structures, such as the SCA, global locks, and group buffer pools, stored in the coupling facility.

1.3.1 Data sharing configurations for high availability and scalabilityWe complete the DB2 data sharing infrastructure picture with data sharing configuration options for PeopleSoft.

There are four basic data sharing options for providing a highly available environment for a PeopleSoft database using DB2 on z/OS:

� Single active DB2 member with passive (inactive) standby member

� Two active DB2 members without passive standby members

� Two active DB2 members each with a passive standby member in opposite LPAR

� Two active DB2 members each with a passive standby member in independent LPAR

Single active DB2 member with passive standby member - option 0The objective of this configuration is availability. Figure 1-3 on page 7 introduces the notion of primary DB2 members, which normally have PeopleSoft application servers attached doing productive work, and standby DB2 members, which are normally in hot standby mode with no attached applications servers.


Figure 1-3 Single active DB2 member with passive standby member

Each PeopleSoft application server goes through DB2 Connect™ to connect to a DB2 member on z/OS. In the event of a planned or unplanned incident which makes a DB2 member unavailable, DB2 Connect recognizes the need to failover, looks for standby information in its setup definitions, and connects the PeopleSoft application server to the standby DB2 member.

This option is chosen most often when high availability is the main concern and the current hardware is sufficient to handle all of the PeopleSoft database requirements. In other words, your PeopleSoft database workload can be handled by one DB2 data sharing member in one CEC.

Note that the LPAR MVS2 may have other non-PeopleSoft work assigned, but the load there should be light enough that the occurrence of PeopleSoft failover will not cause it to suffer degraded performance.

Two active DB2 members without passive standby members - option 1

The objective of this configuration is processor scalability and virtual storage relief; it is not availability. The DB2 member DSN2 performs work on behalf of PeopleSoft in normal operation; it provides access to the additional processing

Data sharing group PROD

CF

PrimaryConnection

StandbyConnection

PrimaryDB2 member DSN1

Lpar MVS1 Lpar MVS2

StandbyDB2 member DSN2

DB2 Connect

Apps Server


power found in MVS2, and it provides an additional address space of 2 GB of storage for DB2 buffer pools, locks, and prepared SQL (Figure 1-4).

Figure 1-4 Two active DB2 members without passive standby members

This option is considered most often when one DB2 member running in one CEC or LPAR cannot handle the PeopleSoft database workload from a CPU capacity standpoint or the PeopleSoft database workload would consume the entire 2 GB of virtual memory in the DBM1 address space. The 2 GB virtual storage limit is true for DB2 releases V7 and earlier, although DB2 V7 enables you to implement dataspaces to take advantage of 64-bit real storage on zSeries machines.

When thinking about configuring DB2 to support more workload, this is most often the first thought that comes to mind. In this configuration, PeopleSoft sysplex failover scenario is set up so that application servers will move (connect) from the failing active DB2 member to the other active DB2 member. If the respective machines supporting these DB2 members are sized just to fit the normal workload, then one should expect degraded performance when all of the PeopleSoft database workload is concentrated on the surviving DB2 member. This degraded performance could come about due to lack of CPU capacity or lack of memory.


CF

PrimaryConnection


Lpar MVS1 Lpar MVS2


DB2 Connect

Apps Server

PrimaryConnection

DB2 Connect

Apps Server


With recent announcements of the new zSeries z990 hardware and the latest version, V8.1, of DB2 software, both the CPU and virtual storage constraints are being lifted. With 64-bit addressing support, it might be possible to revert to the previous option and still support increased PeopleSoft workload.

In general, we do not recommend that this option be used. We recommend option 2 instead. If you decide to use this option anyway, then we remind you to give careful consideration to sizing the hardware properly and configuring Workload Manager (WLM). If you require the same level of performance, no matter what state the system is in, then each system should have enough CPU and memory capacity reserved to handle the maximum additional workload on each system. This standby capacity can be very costly if using the resources just for one PeopleSoft system. Fortunately, one of the great strengths of z/OS on zSeries is the capability to support multiple workloads simultaneously. This is where WLM is important. WLM enables you to assign importance to each workload. So in the event of a failover of workload to one surviving DB2 member, WLM can be configured to ensure that the PeopleSoft workload receives priority over the other workload, even if it is non-production PeopleSoft workload.

A real-life customer example of Option 1 is described in Appendix A, “PeopleSoft data sharing configuration (option 1): customer example” on page 111.

Two active DB2 members, each with a passive standby member in opposite LPAR - option 2

Data sharing option 2 is a realistic implementation for a large PeopleSoft installation to allow database availability. With option 2, standby data sharing members are provided for each of two primary members, as shown in Figure 1-5 on page 10.


Figure 1-5 Two active DB2 members, each with a passive standby member in opposite LPAR - Option 2

Option 2 is really just a variation of option 0. In both options you have an active and standby DB2 member, but option 2 just has more pairs of active and passive members. This option is recommended (or even required) to support any PeopleSoft requirement that exceeds the capacity of a single machine.

Another use of this option would be to isolate PeopleSoft business components from each other. This is the logical extension of having separate application servers to run specific PeopleSoft modules.

As in the other options, it is possible that the MVS1 and MVS2 LPARs are performing other work, as long as the load is light enough to enable adequate performance if activation of one of the standby DB2 members occurs.

Two active DB2 members, each with a passive standby member in independent LPAR - option 3

Option 3 represents the option 2 solution carried to the next level. Not only are there standby data sharing members, there are also standby LPARs in which the DB2 members reside (Figure 1-6 on page 11).

StandbyConnection


CF

PrimaryConnection


Lpar MVS1 Lpar MVS2

Apps Server

PrimaryConnection

Apps Server




StandbyConnection

DB2 Connect DB2 Connect


Figure 1-6 Two active DB2 members, each with a passive standby member in independent LPAR - Option 3

At first glance it may appear that additional LPARs for the standby function is a heavy price to pay for availability, but most installations that incorporate option 3 assign the standby function to one of the LPARs in their test-to-production landscape. That is, the normal function of MVS3 or MVS4 could be as a test or quality assurance machine, functions that can be interrupted if an availability situation arises.

However, the use of machines in the test-to-production landscape is not without risk; they could be using a level of z/OS or DB2 for z/OS that has not yet been put into production. Failover scenarios could then encounter situations with this new code that have software failures, operational differences, or performance impacts that are unexpected.

1.3.2 How many data sharing groups?A typical PeopleSoft landscape consists of a development system, a quality control system, a stress test system, and a production system. Optionally, one

StandbyConnection


CF

PrimaryConnection


Lpar MVS1 Lpar MVS4

Apps Server

PrimaryConnection

Apps Server




StandbyConnection

DB2 Connect DB2 Connect

Lpar MVS3 Lpar MVS2


might decide to have a training system, a technical sandbox, or a production support system.

It is common these days for businesses to roll out the other PeopleSoft technology components such as PeopleSoft HR, Financials, and so on to support the next generation of PeopleSoft solutions. So it is quite common to have separate PeopleSoft landscapes for each PeopleSoft component.

Whatever PeopleSoft components or solutions you implement, the total number of PeopleSoft systems to build and maintain can add up quickly. It seems that every group involved in implementing a PeopleSoft system or solution wants their very own system to work with. Of all of those systems, how many should be configured for high availability and performance using data sharing?

As you are reading this book, you have already decided that your PeopleSoft production system should be configured to be highly available and/or scalable. Therefore, the production system must be configured at the very least to run in DB2 data sharing mode.

What about the non-production PeopleSoft systems? The answer is not so easy. It depends on your service level agreement (SLAs). Some SLAs require that even the development system be highly available. It is very costly to have developers who cannot work because the system is unavailable. Whatever your SLA, we recommend that you configure at a minimum one other PeopleSoft system for DB2 data sharing in your promote-to-production landscape. This system is where you would test your code or configuration changes to verify that there are no problems related to running them in your data sharing setup. Your production system is not the place to learn that your changes do not work with DB2 data sharing.

Which non-production system should be configured for data sharing? It depends on how soon or late you want to test your changes with data sharing. Applications will run just fine with data sharing when doing single-user or component-level tests. The story might be quite different for stress tests. As the number of users running against different systems increase, you might have a bigger potential for resource contention, so we recommend that your other data sharing system is either your quality control system or your stress test system if you have one.

We recommend further that you consider having at least one additional data sharing system in each of your PeopleSoft landscapes where your business needs require that you have high availability for your production system. While each PeopleSoft component shares common technology, there does exist non-common functionality. A more important thing to keep in mind is specific landscape configuration work. So it is recommended to have one additional data sharing system per PeopleSoft Landscape.


So far we have concentrated on figuring the number of data sharing systems to make sure that the application changes and PeopleSoft system changes do not cause any problems with data sharing. What about the infrastructure changes like coupling facility changes? What system should the infrastructure group use to test their changes? The infrastructure group should consider building a Parallel Sysplex with data sharing system that is independent of the production and non-production PeopleSoft systems. It is sufficient to have one Parallel Sysplex for the infrastructure group. There is no need nor benefit to have one technical sandbox system per PeopleSoft landscape. This approach, while consistent, would be cost-prohibitive.

1.3.3 How many data sharing members?When you have decided that you need DB2 data sharing, the next question is how many data sharing members are required for each highly available system. The answer to this question depends on the data sharing option you are implementing. The option you choose depends on the sizing estimate for your proposed production system or systems.

For an option 0 production system, you need only two data sharing members per data sharing group. The primary data sharing member does all of the work, and the secondary data sharing member is a standby only. We call this passive data sharing. This option is valid as long as your workload does not exceed the PeopleSoft rating of the zSeries box or production LPAR. One fact about this configuration that is difficult to live with is that you have a second zSeries box or LPAR of at least equal CPU capacity, memory capacity, and I/O bandwidth sitting there using electricity, but not doing any active work. So that leads us to option 1.

For an option 1 production system you have active workload distributed between two or more members of a data sharing group. Assume that you are configuring a two-way data sharing system: If one system fails or becomes unreachable, the workload will be moved to the surviving data sharing member. If you want the system to perform in failover mode with same level of throughput as in non-failover mode, then you must ensure that there will be sufficient extra CPU capacity, memory capacity, and I/O bandwidth to handle the failover work in one zSeries box or LPAR. Basically, you must double the capacity of each zSeries box. A zSeries box rarely fails, so it may not be so important to have all of that extra capacity ready for a failover situation that will rarely happen. In the DB2 V7.1 and earlier releases, keep in mind that there is a 2 GB limit on the addressable memory in the DBM1 address space. Every PeopleSoft work process that connects to the surviving data sharing member will consume from 1.3 to 2.5 MB (or even higher). So you must plan for the maximum number of RRSAF connections per data sharing member.


Another possible option 1 production system could have three data sharing members in a group. If one data sharing system fails with this version of option 1, you have the option to redistribute the workload to one of the surviving members or to redistribute the workload evenly to the surviving members. When making this decision, the main choices are to minimize DB2 inter-systems interest or not overallocate DB2 DBM1 virtual storage. To minimize DB2 inter-systems interest, ideally you would move all of the workload from the failing data sharing member to one of the surviving data sharing members. This might lead to overallocation on DBM1 virtual storage. On the other hand, to prevent possible overallocation of memory, you could evenly distribute the workload among the surviving members. However, this might increase DB2 inter-systems interest between the two surviving members. Which is the best choice to make? It is better to have the system available providing reduced throughput than to overallocate DBM1 virtual storage and risk another abend that would reduce throughput even more. Therefore, we recommend for option 1 systems redistribution of the workload evenly among the surviving data sharing members.

In order to minimize DB2 inter-systems interest and prevent the overallocation of memory in a failover situation, we recommend that you implement option 2 instead of option 1. Option 2 eliminates DB2 inter-systems interest, because all workload from the failing DB2 data sharing member would move to a standby member. Also, no part of the surviving primary data sharing members would be affected. This standby member could be started, ready, and waiting in the same LPAR as the primary or in another LPAR. Option 2 eliminates the possibility of overallocation of DBM1 virtual storage, because the failover happens to an empty data sharing member that is configured exactly the same as the primary. In the event that only some of the PeopleSoft work processes fail over to the standby data sharing member, a corresponding number of PeopleSoft work processes would be eliminated from the primary data sharing member. There would be no overallocation of virtual storage, but there could be inter-systems interest, but only for that portion of workload running on those data sharing members.

We recommend that you configure your non-production data sharing system or systems exactly the same way as your production system.

The notion of standby DB2 members was introduced for several reasons: less disruptive to surviving DB2 members (no washing of the buffer pools or dynamic statement caches) and it reduces the need to ensure that there is sufficient DBM1 virtual storage to absorb the movement of a large number of PeopleSoft work processes (and hence DB2 threads). There are two types of companies doing PeopleSoft-DB2 data sharing: those who are motivated primarily by near-continuous availability (secondarily the ability to level workload across the multiple CECs required for no single point of failure) and those who have both a scalability and availability objective.


The first type typically implements option 0 and frequently is characterized by having many PeopleSoft production systems.

The second type generally implements option 2. Many of the companies pursuing both scalability and high availability have three or more CECs in their sysplex. A typical configuration in a three-CEC sysplex would have a primary DB2 on each CEC with a standby DB2 member residing in each production LPAR that contains a primary DB2. In the event of planned or unplanned loss of one of the production environments (such as CEC1), half of the application servers will reconnect to the standby DB2 member on CEC2, and half will connect to the standby member on CEC3. Although this is more complex than simply moving all of the application servers to one standby DB2, it does offer workload management benefits. Assuming that each of the three primary DB2s were servicing one-third of the total workload, half of this (one-sixth of the total workload) will be moved to each standby member in the surviving CECs. When coupled with the Workload Manager's ability to differentiate priorities (goals, importance, velocity) based on PeopleSoft work process type, very high interactive service levels can be maintained with minimized disruption to the surviving CECs. This enables us to minimize the purchase of extra capacity to support failover.


Chapter 2. Performance considerations: implementing PeopleSoft with Parallel Sysplex data sharing

Our rationale for approaching this subject is to build a bottom-up case for good PeopleSoft performance. A well-performing sysplex is the base for a well-performing DB2 data sharing system, and a well-performing DB2 data sharing system is the base for a well-performing PeopleSoft system.

In this chapter we discuss performance issues in the order you would approach them at planning and implementation time:

� Sysplex performance

� Data sharing performance

We primarily focus on methodology and refer you, when necessary, to publications where you can find more detailed discussion on the subjects covered.

2

© Copyright IBM Corp. 2004. All rights reserved. 17

2.1 Sysplex performance This section discusses sysplex performance and monitoring issues relevant to PeopleSoft data sharing environments. A performing sysplex that is well set up and healthy is the basis for good performance in your DB2 data sharing system.

To ensure that your tuning efforts are producing the expected results, you should keep a baseline for a before-and-after comparison. This baseline may consist of:

� Coupling Facility processor utilization� Average synchronous and asynchronous service time� Percentage of changed requests

We describe the following:

� CF performance monitoring and tuning� Sysplex workload monitoring� WLM policies� RMF™ Monitor III� RMF Post Processor reports� RMF spreadsheet

2.1.1 CF performance monitoring and tuningIt is important to have a basic understanding of CF performance tuning concepts in order to have a properly tuned DB2 data sharing group. Three items should be monitored:

� CF processor monitoring� CF storage utilization� CF link performance

Monitoring CF utilization and performance can be achieved by using RMF Spreadsheet Reporter, which is described in 2.1.5, “RMF Spreadsheet Reporter” on page 28.

CF processor monitoringCF processor utilization must be kept below 50% because synchronous service time starts to degrade with higher utilization. High sync service time may have a performance impact because the requestor’s processor has to wait for the request to complete. Besides reducing CF efficiency, it also increases CPU utilization in the z/OS side.

Our recommendation is to keep CF utilization under 30%, if you are in a single CP Coupling Facility, in case a failover happens and one CF has to absorb the workload from a failing CF.


To obtain information about CF utilization, run an RMF Post Processor report with option SYSRPTS(CF).

CF storage utilizationCF level 12 introduces 64-bit mode support and removes the 2 GB limitation that existed for some DB2 structures. CF level 12 provides the following advantages:

1. 64-bit CFCP with CFLEVEL12 and OS/390 R10 to overcome 2 GB control storage limit and provide for very large LOCK1 structures (because of modified record list requirement)

2. 64-bit real with OS/390 R10 and any z/OS operating system to support dataspace buffer pools and provide VSCR above line in DBM1

3. 64-bit virtual support with z/OS R3 and DB2 for z/OS V8 to provide even more VSCR

4. Reduced data sharing overhead using WARM/RFCOM with z/OS R4, CFLEVEL12, and DB2 V8

5. SLPIT Recovery (BACKUP SYSTEM utility and RESTORE SYSTEM utility without LOGONLY option) with z/OS R5 and DB2 V8

Special care also must be taken in anticipation of failover scenarios. The surviving CF must have enough capacity to absorb the structures coming from a failing CF.

CF link performanceService time is monitored from the moment an exploiter issues a CF request to the moment z/OS receives the command return. Service time is recorded in microseconds for each structure used by each system. As CF links can be shared between two or more LPARs in the same CPC, it is possible that some contention exists.

When such a contention occurs, one request is processed and the other is rejected. This situation gets registered as Path busy. It is important to check the total percent of delayed requests in the Subchannel Activity section from an RMF CF report.

If the total path busy is greater than 10% of all requests, this indicates link contention, consider dedicating CF links to the CF from each partition or adding more shared links to the CF.

Each CF link is capable of supporting a certain number of concurrent CF operations. The maximum number of concurrent CF operations is the total number of subchannels (buffers) defined to a CF.

Chapter 2. Performance considerations: implementing PeopleSoft with Parallel Sysplex data sharing 19

When all subchannels are busy performing operations to the CF, then newly incoming requests are delayed. This is known as a subchannel busy condition. For SCA and Group buffer pool requests, z/OS will queue these delayed requests and process them in first-in-first-out order as soon as one of the active operations ends. For a lock structure, z/OS will generally spin and wait for one of the active requests to finish and then re-issue the request.

In summary, for SCA and Group buffer pools, increased subchannel busy may lead to request queuing, thus affecting performance. For the Lock structure, high subchannel busy leads to higher CPU utilization and diminished CF efficiency.

If the total subchannel busy is greater than 10% of all requests, you may have to add another CF link between the CF and the z/OS partition.

We recommend monitoring service times. The zSeries family of processors introduced three additional links: ISC-3, ICB-3, and IC-3. These links must be configured in Peer Mode. Peer Mode supports CF between z900 servers and provides both sender and receiver capability on the same link. Peer links provide up to seven expanded buffer sets (subchannels), which in reality become up to 14 buffer sets as both receiver and sender traffic travel in the same link.

Each link type has its own characteristics. They facilitate improved channel efficiency and service times in the processing of CF requests. But it is beyond the scope of this book to provide more in-depth details on CF links.

Service time varies according to the hardware involved. Figure 2-1 shows the guidelines for service times for the z/900 series:

Figure 2-1 Service times for z/900 series

Starting with z/OS 1.2, IBM has introduced a new algorithm to decide whether to convert synchronous requests to asynchronous requests. This new function monitors CF service times for all (CACHE, LIST, and LOCK) structures and, based on new internal thresholds, determines which requests would be more efficiently executed asynchronously. Thresholds are different for simplex and duplex requests and for Lock and non-Lock requests.

Synchronous service time: GBPs LOCK1, SCAICP 30 mics 20 micsCFP 65 mics 40 mics

Asynchonous service time: GBPs LOCK1, SCAICP 120 mics 100 micsCFP 150 mics 120 mics


For more insight, review Washington System Center Flash # 10159. You can download it from:

http://www-1.ibm.com/servers/eserver/zseries/library/techpapers/pdf/gm130103.pdf

CF signalling path length monitoringz/OS systems within a sysplex must constantly pass messages to each other. These signalling paths are heavily used when IRLMs have to negotiate locks in a data sharing environment.

When Global Lock contention is detected, Global Lock Management via XES/IRLM uses message passing and is heavily dependant on fast XCF communications. Global Lock Management is synchronous with respect to the application. The lack of enough XCF signaling resources to accommodate Global Lock contention management traffic can be detrimental to DB2 performance. It is best to overconfigure XCF signaling resources.

We have observed that low-weight LPAR partitions can get worse performance for anything dependent on asynchronous operations and XCF messaging (such as Global Lock contention). There are several spots they can slow down:

� The z/OS dispatcher must run to see an asynchronous completion. The LPAR may not have any logical CPs for MVS1D dispatched when the asynchronous operation completes, so MVS1D is very late in noticing.

� Several SRBs must run to complete the asynchronous operation, post XCF, and then wake up IRLM. LPAR can take away the logical processor running any of this work any time. This could also slow down lock contention on other partitions, as they sometimes have to wait for MVS1D to respond to them

Transfer time between LPARs can only be seen by issuing the commands D XCF,PI,DEV=ALL,STATUS=WORKING and D XCF,PI,STRNM=ALL. The recommended transfer time is less than 2000 microseconds (or 2 milliseconds). These IRLM negotiations are synchronous, so keeping them under 2000 microseconds is very important.

It is recommended that these two commands be issued automatically every hour.

To monitor signalling paths, you must run an RMF postprocessor with REPORTS(XCF) control card. For XCF tuning guidelines, refer to IBM Washington Center Flash # 10011 at:

http://www-1.ibm.com/servers/eserver/zseries/library/whitepapers/pdf/availchk_parsys.pdf

This flash provides a step-by-step road map to ensure that transfer time between LPARs is less than 2000 microseconds.


http://www-1.ibm.com/servers/eserver/zseries/library/techpapers/pdf/gm130103.pdfhttp://www-1.ibm.com/servers/eserver/zseries/library/whitepapers/pdf/availchk_parsys.pdf

Figure 2-2 shows an example of an RMF postprocessor output from option REPORTS(XCF).

Figure 2-2 RMF postprocessor output from REPORTS(XCF)

2.1.2 WLM policiesThe purpose of Workload Manager (WLM) is to balance the available system resources to meet the demands of z/OS subsystems and applications based on user-defined performance objectives. WLM provides a mechanism for managing workload distribution, workload balancing, and distribution resources to competing workloads. It manages the workload of the entire z/OS system. This section focuses on some of the considerations for the PeopleSoft application.

WLM is a critical success factor. We strongly recommend that you configure a WLM policy specific to PeopleSoft to help you manage the performance of your PeopleSoft system and achieve good overall system performance for your sysplex.

WLM service definition contains all of the information about the installations needed for workload management processing. There is one service definition for the entire sysplex. The service level administrator specifies the service level definitions for the entire sysplex. The service level administrator sets up policies within the service definition to specify the goals for work. A service level administrator must understand how to organize work and be able to assign it performance objectives.

Is it important that the PeopleSoft applications be included in the WLM service definition. The PeopleSoft applications should be viewed as a multi-application subsystem within the z/OS subsystems and applications. Because of the variety of workloads that can be run within PeopleSoft, proper performances should be established within the PeopleSoft suite. In general, you should consider giving

----- BUFFER ----- ALL TO TRANSPORT BUFFER REQ % % % % PATHS REQ SYSTEM CLASS LENGTH OUT SML FIT BIG OVR UNAVAIL REJECT 3W02 DEFAULT 16,316 119,266 100 0

the online PeopleSoft applications higher performance goals than the PeopleSoft batch workloads.

When the PeopleSoft application suite performance goals are established, the PeopleSoft workload definition must be blended with the total z/OS workload.

Another consideration for the WLM service definition is to define reporting classes for each PeopleSoft process type. This provides the greatest granularity for monitoring purposes. The Resources class definition included in Example 2-1 shows how this can be accomplished.

Example 2-1 Sample setup for WLM and PeopleSoft V8

1 SI PRODDB2 DDFMED PRODDB2 2 UI PRHRID DDFMED PSHRDFLT 3 . CI . cr* PSOFTSQP PRHRCR 3 . CI . CR* PSOFTSQP PRHRCR 3 . CI . db2b* PSOFTSQP PRHRDB2B 3 . CI . javaw* PSOFTSQP PRHRJAVW 3 . CI . PSAE* PSOFTSQP PRHRPSAE 3 . CI . psae* PSOFTSQP PRHRPSAE 3 . CI . PSCR* PSOFTSQP PRHRPSAE 3 . CI . pscr* PSOFTSQP PRHRPSAE 3 . CI . PSDAEMON PSOFTSQP PRHRDAEM 3 . CI . psdaemon PSOFTSQP PRHRDAEM 3 . CI . PSAPPSRV PSAPPSRP PRHRAPPS 3 . CI . PSQCKSRV PSAPPSRP PRHRQCKS 3 . CI . PSSAMSRV PSAPPSRP PRHRSMAS 3 . CI . PSQRYSRV PSAPPSRP PRHRQRYS 3 . CI . PSBRK* PSAGENTP PRHRBRK 3 . CI . PSMSG* PSAGENTP PRHRMSG 3 . CI . PSPUB* PSAGENTP PRHRPUB 3 . CI . PSSUB* PSAGENTP PRHRSUB 3 . CI . PSPRCSRV STCMD PRHRSCHD 3 . CI . PSDSTSRV PSAGENTP PRHRSCHD 3 . CI . PSQE* PSOFTSQP PRHRQRY 3 . CI . psqe* PSOFTSQP PRHRQRY 3 . CI . PST* PSAPPSRP PRHRPST 3 . CI . pst* PSAPPSRP PRHRPST 3 . CI . PSS* PSAPPSRP PRHRTR2 3 . CI . pss* PSAPPSRP PRHRTR2 3 . CI . PSDM* PSOFTSQP PRHRPSDM 3 . CI . dsdm* PSOFTSQP PRHRPSDM 3 . CI . PSD* PSAPPSRP PRHRTRD 3 . CI . SQR* PSOFTSQP PRHRSQR 3 . CI . sqr* PSOFTSQP PRHRSQR 3 . CI . PSIDE* PSAPPSRP PRHRIDE 3 . CI . pside* PSAPPSRP PRHRIDE 3 . CI . PSNV* PSOFTSQP PRHRNVS


3 . CI . psnv* PSOFTSQP PRHRNVS 3 . CI . PS* PSOFTSQP PRHRDDF

2 UI PRFIID DDFMED PSIFDFLT 3 . CI . cr* PSOFTSQP PRFICR 3 . CI . CR* PSOFTSQP PRFICR 3 . CI . db2b* PSOFTSQP PRFIDB2B 3 . CI . javaw* PSOFTSQP PRFIJAVW 3 . CI . PSAE* PSOFTSQP PRFIPSAE 3 . CI . psae* PSOFTSQP PRFIPSAE 3 . CI . PSCR* PSOFTSQP PRFIPSAE 3 . CI . pscr* PSOFTSQP PRFIPSAE 3 . CI . PSDAEMON PSOFTSQP PRFIDAEM 3 . CI . psdaemon PSOFTSQP PRFIDAEM 3 . CI . PSAPPSRV PSAPPSRP PRFIAPPS 3 . CI . PSQCKSRV PSAPPSRP PRFIQCKS 3 . CI . PSSAMSRV PSAPPSRP PRFISMAS 3 . CI . PSQRYSRV PSAPPSRP PRFIQRYS 3 . CI . PSBRK* PSAGENTP PRFIBRK 3 . CI . PSMSG* PSAGENTP PRFIMSG 3 . CI . PSPUB* PSAGENTP PRFIPUB 3 . CI . PSSUB* PSAGENTP PRFISUB 3 . CI . PSPRCSRV STCMD PRFISCHD 3 . CI . PSDSTSRV PSAGENTP PRFISCHD 3 . CI . PSQE* PSOFTSQP PRFIQRY 3 . CI . psqe* PSOFTSQP PRFIQRY 3 . CI . PST* PSAPPSRP PRFIPST 3 . CI . pst* PSAPPSRP PRFIPST 3 . CI . PSS* PSAPPSRP PRFITR2 3 . CI . pss* PSAPPSRP PRFITR2 3 . CI . PSDM* PSOFTSQP PRFIPSDM 3 . CI . dsdm* PSOFTSQP PRFIPSDM 3 . CI . PSD* PSAPPSRP PRFITRD 3 . CI . SQR* PSOFTSQP PRFISQR 3 . CI . sqr* PSOFTSQP PRFISQR 3 . CI . PSIDE* PSAPPSRP PRFIIDE 3 . CI . pside* PSAPPSRP PRFIIDE 3 . CI . PSNV* PSOFTSQP PRFINVS 3 . CI . psnv* PSOFTSQP PRFINVS 3 . CI . PS* PSOFTSQP PRFIDDF

The example shows that HR and FI are separated by a UI classification rule. This type of rule can also be used to separate production HR from test, dev, and so forth.

Also notice that service classes are shared by many CIs but report classes are mostly unique.


Finally, there are cases where CIs come through in lower case; therefore, both cases have to be specified.

Here are some samples for the service class definitions, assuming that the service coefficients are set to a multiplier of 1:

PSAGENT - single service class, importance 3 or 4, velocity of 30PSAPPSRP - two period service class:

period 1 - importance 1, duration 500, velocity 50period 2 - importance 3, velocity 30

PSOFTSQP - two period service class:period 1 - importance 3, duration 10000, velocity 40period 2 - importance 4, velocity 30

The key to establishing the proper WLM service definition is to understand the entire workload on the system, then to monitor the system through detailed reporting class definitions and make the necessary adjustments as the workload demands change.

2.1.3 RMF Monitor III - Work Delay MonitorThe RMF Work Delay Monitor, otherwise known as RMF Monitor III, provides very useful reports for identifying delays in the system components that have an impact on your PeopleSoft workload performance:

� CF activity overview

From your ISPF RMF Monitor III menu, select S SYSPLEX → 5.

This displays an overview of your coupling facility activity. Check that the CPU activity is no higher then 50% and that there is free storage.

� CF paths

From your ISPF RMF Monitor III menu, select S SYSPLEX → 6.

This displays availability and performance of the paths to the CF. Check that you have all defined paths available and that the delay % is not too high.

� Delay Report

From your ISPF RMF Monitor III menu, select 1 → 4.

This takes you to the Delay Report. A shortcut to this report is to enter DLY from the command line on any RMF Monitor III screen.

Delay Report is a very useful report for identifying delays to your PeopleSoft system workload components.


The two main types of delay you are likely to see are PRC and DEV:

– PRC

PRC indicates a delay for processor resources. If you have not set up your WLM service classes correctly, then you may have less important work getting preference for CPU resources over and above your DB2 or PeopleSoft workloads.

One cause of PRC delays might be that the LPAR is reaching its allowable CPU limit. You should check the PR/SM™ weightings and the CPU activity for the machine. The CPU activity can be checked using the RMF post processor reports. You can also see the machine CPU activity in real time from the RMF Monitor III. To see the machine CPU activity from Monitor III do the following:

From your ISPF RMF Monitor III menu, select 1 → 3.

This option shows the CPU activity for the whole machine and for the LPARs defined on the machine.

– DEV

This indicates a delay for a DASD or tape. For your DB2 address spaces, you may see DASD device delays. This can happen for several reasons, the most common being that there are too many active tablespace datasets on a single volume. This can be related directly to your PeopleSoft database response times. If you have high access times for a table, as identified in your PeopleSoft single record statistic analysis or your PeopleSoft SQL trace analysis, then this may be caused by device delays in DB2 for the volume.

In the DB2 system overview, you might also see high I/O suspend times.

You can investigate further by selecting the resource device delay report as follows:

• From your ISPF RMF Monitor III menu, select 3 → 2.

This shows disk volumes that are causing delays to the current workload. You can get further detail on the datasets that are being accessed on the volume.

• From your ISPF RMF Monitor III menu, select O → 4 → Report=DSNV.

Now enter the volser of the volume you want more details about.

Return to the ISPF RMF Monitor III menu.


• From your ISPF RMF Monitor III menu, select 3 → 3B.

This produces the detail dataset delay report for that volume. You will see which datasets and therefore which tablespaces are on the volume that are being accessed.

Device delays can be reduced with the Parallel Access Volume (PAV) facility. PAV, in either dynamic or static form, can make a big difference in terms of I/O response times by significantly reducing IOSQ time while queuing on the UCB. PAV involves creating aliases for the UCB to reduce queuing and to push it down where possible to DASD controller. Use of PAV can significantly reduce (not eliminate) the need for careful dataset placement.

If you cannot use PAV, then you should consider reducing I/O contention by careful dataset placement. Even if you have implemented PAV you should still think about controlling dataset placement to achieve optimal performance and to use your DASD resources to their full potential. Dataset placement can be achieved by implementing SMS policies to ensure separation of data across volumes.

Another way to reduce I/O delays is to partition the tablespace to spread the I/O across volumes. Even if you have PAV you should still consider controlling the dataset placement for the partitions by using SMS policies or manual methods. You should keep in mind that OS/390 z/OS queues I/O activity at the UCB for the volume, despite that fact that you may be using new technology DASD hardware.

2.1.4 RMF post processor reportsRMF post processor reports provide historical performance data for your PeopleSoft data sharing environment. This section highlights some of the reports that you can use to analyze performance problems in your PeopleSoft data sharing environment. This is not an in-depth discussion of RMF reporting, but for PeopleSoft consultants who want to investigate performance in a data sharing environment, it is useful to be aware of the additional OS/390 and z/OS reporting facilities.

Further documentation on running these reports and reading the output can be found in the RMF Report Analysis manual for your operating system release.

� CPU Activity Report

This report is useful for investigating CPU resource delays in your system. It can provide historical information about logical and physical processor utilization in your sysplex.


� Coupling Facility Activity Report

This report provides useful information about each coupling facility attached to your sysplex. If your previous investigations lead you to believe that the performance problems are due to poor data sharing performance, then this report provides useful information about the activity in the coupling facility.

� XCF Activity Report

This report provides information about the activity of your cross-system coupling facility. This is useful if you suspect that poor performance may be due to poor data sharing performance.

� Device Activity Report

In this report one of the important items is to check whether PAVs are being distributed properly as needed. Either dynamic or static form can make a big difference in terms of I/O response times by significantly reducing IOSQ time on the UCB queuing.

PAV involves creating aliases for the UCB to reduce queuing and to push it down where possible to the DASD controller. Use of PAV can significantly reduce (not eliminate) the need for careful dataset placement.

2.1.5 RMF Spreadsheet ReporterRMF Spreadsheet Reporter is a set of applications comprised of Excel macros that can be downloaded to your workstation. RMF Spreadsheet Reporter extracts information from a server mainframe RMF Postprocessor report and presents this data graphically. This tool can be downloaded for free from:

http://www-1.ibm.com/servers/eserver/zseries/zos/rmf/rmfhtmls/rmftools.htm

RMF Spreadsheet Reporter consists of four applications:

� Collector� Extractor� Converter � Several spreadsheets

After installing the product, you might need to customize the provided skeleton JCL in C:\RMFPP\PROGS\rmfpp.jcl. In some cases, you may want some customization that is not part of Collector.

RMF Spreadsheet Reporter enables you to submit an RMF Postprocessor job from your workstation into a server mainframe through TCP/IP.


http://www-1.ibm.com/servers/eserver/zseries/zos/rmf/rmfhtmls/rmftools.htm

CollectorThis module enables you to provide all information to build the job using the previously mentioned skeleton JCL. It enables you to choose the date and time, duration interval, and type of reports, and at job submission time provides two options for receiving the listings, Immediate and Deferred:

� Immediate

Immediate starts your job and waits for its conclusion. If you are reading SMF buffer data from RMF, it is likely that your job will end in less than 10 minutes because it reads data that requires no sorting. Be careful because the default FTP value for JESPUTGETTO is 600 seconds. This parameter specifies how long an FTP-job-submitter can wait on the job until it is finished.

� Deferred

Deferred is more appropriate because you need to read from SMF tape files and sort it. Your job is submitted and, when it ends, the output can be sent to your workstation via FTP.

Extractor When the result set from your batch job is sent to your workstation, you can run the Extractor. It creates a Report-Work-Set from the downloaded file.

Converter The Converter takes the output from the Extractor and converts the RMF reports to a spreadsheet format.

SpreadsheetsThe Excel macros are the core of RMF Spreadsheet Reporter. There are several type of macros. For our tuning purposes, we recommend:

� Summary report

A graphically represented summary report for CPU utilization and DASD I/O rate.

� DASD activity report

Graphically represented detailed I/O report.

� Workload Activity Trend in goal mode report

Graphically represented detailed report on Service Class and so on.

� Coupling Facility trend report

Detailed data for all CF structures, graphical reports on CF utilization, number of requests, service times for all LPARs, percentage of changed requests due to path busy or subchannel busy. Highly recommended.


RMF Spreadsheet Reporter is a great jump forward in assisting the whole mainframe server (such as z/OS and DB2) in the areas of creating baselines, tuning, and tracking performance.

2.2 Data sharing performance This section discusses DB2 data sharing performance and monitoring issues that are relevant to PeopleSoft implementations in a sysplex environment. A well-configured data sharing environment is the basis for good performance in your PeopleSoft system. We describe the following:

� Local buffer pool performance� Group buffer pool performance� Dynamic SQL Cache� Lock performance� DB2PM statistics batch report� Table space partitioning� Managing DBM1 virtual storage� Configuring DB2 sort pool and workfiles� DB2 backup policies; avoiding disruptive backups

Having a baseline is also very important. This is the only way to ensure that improvements derived from DB2 system tuning can be quantified.

When we tune buffer pools our final objective is to reduce the number of pages read in or I/O traffic. I/O savings can be not only quantified but also compared to MIPS consumption through CPU time savings. Buffer pool hit ratios may be good performance indicators, but I/O traffic is a much better tracking metric.

We have used such a baseline for trend reporting as well. The suggested baseline can be a 24-hour period updated daily. At the beginning of the week, this baseline, which can be in a spreadsheet, can be published to all involved and management. It can be a powerful decision support tool.

The baseline can be comprised of:

� Total number of synchronous I/O reads for all local buffer pools

� Total number of pages read sequentially for all local buffer pools (pages read via seq. prefetch + pages read via list prefetch + pages read via dynamic prefetch)

� Global and Local Dynamic SQL Cache hit ratios

� Global locking contention rates: Real and False

� Average CPU consumption


The first three items come from a DB2PM Statistics report, and the rest can be obtained from an RMF Postprocessor CPU report.

In large environments, browsing different performance reports daily and acting in a pro-active manner can be time-consuming. But with such a spreadsheet, we usually have an idea of the overall health of our DB2 systems.

For instance, any 20% increase in I/O traffic (sync or sequential) for more than a day or two and even if the average DB response time is not being affected, warrants a more detailed investigation.

This baseline is also good to help you spot changes in the application behavior that you may not be aware of otherwise. For example, sometimes new SQL challenged code is promoted into production without the DB2 systems group being aware of it. You should be able to detect it, as I/O traffic is increased by this change.

2.2.1 Local buffer pool performanceOne of the most important points when designing DB2 buffer pools for a PeopleSoft environment is to group similar objects according to their characteristics, such as:

� Large workset, highly updated, random access� Small workset, highly updated, random access� Large workset, little updated, random access� Small workset, little updated, random access� Highly updated, sequential access� Little updated, sequential access

When tuning DB2 buffer pools the primary goal is to reduce or eliminate I/O by finding needed pages in the local buffers and to minimize synchronous single page I/O.

The recommendations for buffer pool tuning are geared toward local buffer pools with a mixed load of random and sequential pages.

You can find the lowest I/O rate for a buffer pool through an estimated page residency time.

If you were to plot buffer pool miss rates versus buffer pool size, the result would be a queuing (exponential) curve. The objective is to avoid getting below the


knee of the cursor, and the recommendation for VP+HP is to get you well beyond the knee and onto the flat part of the curve. The two general rules are:

1. Pages should be resident in the virtual pools for at least a minute.

or

2. Pages should be resident in the combination of virtual and hiperpools for approximately 15 minutes.

Take several reasonably large, typical sample intervals of 15 minutes each and perform these two calculations:

1. Vpool residency = (# of vpool buffers / hiperpool writes per second)

The result should be approximately 60 seconds. This formula can tell how long a page can be in a virtual pool before it is written to a hiperpool.

If the average page residency is much greater than 60 seconds, you can cut back on vpool buffers and give more to hiperpool buffers (at least two hiperpool buffers for each vpool). To get the hiperpool writes, add up synchronous and asynchronous data mover hiperpool writes. This result tells you the number of pages moved out of the vpools by the least recently used algorithm.

2. Total (vpool + hpool) residency = (# of vpool + hpool buffers / pages read per second)

This combined result should be approximately 15 minutes, assuming you have vpools and hiperpools.

If you are not using hiperpools, just calculate residency by using the formulavpool residency = (# of vpool buffers / # of pages read).

Pages read means the total number of pages read via sequential prefetch + number of pages read via list prefetch + number of pages read via dynamic prefetch.

2.2.2 Group buffer pool performanceIn this section we discuss:

� Caching pages in the group buffer pool (GBP)� Calculating GBP hit ratios� Castout efficiency

Caching pages in the GBPThere are mainly two requirements for a page to be cached in the GBP:

� The pages must have been updated in a local buffer pool.


� There must be inter-DB2 interest in the page and at least one member has read/write interest.

When a page set or partition becomes GBP-dependent, all updated pages in the local buffer pool are moved to the corresponding GBPs. Updated pages in the local pool are not always written synchronously to the GBP. Prior to V8, all GBP page writes were CF synchronous operations. Pages were written synchronously with respect to the application using force at commit protocol at commit. Pages were written asynchronous to the application using CF synchronous operations due to VDWQT/DWQT thresholds being reached, or system checkpoint coming due, or as part of page P-lock negotiation. Updated pages were also written synchronously during index leaf page split. In V8, DB2 uses new WARM CF commands (asynchronous CF command, multiple pages). Index leaf page split will also exploit WARM, and the number of GBP writes will be reduced to two.

Changed pages can be written to the GBP before the updated transaction is committed when:

� One of the local buffer pool thresholds (DWQT or VDWQT) is reached.

� The local buffer pool is short of available pages.

� A system checkpoint is issued. Such an event can clean out a local buffer pool before the commit.

� The same page is required for update by another system. This page writing into the GBP is part of page P-lock negotiation.

When such pages are written to the GBP, all copies of those pages that are cached in other members’ buffer pools are invalidated. This process is called cross-invalidation (XI). Next time the application references those pages, they must be loaded from the GBP (or DASD) in a synchronous request.

Another main point to consider is Directory Reclaims. Group buffer pools contain directory entries that are used to maintain information about all pages in all local buffer pools.

Thus, having sufficient directory entries in the GBP optimizes cache structure usage.

However, without sufficient directory entries in the GBP, a new reference to a page in a local buffer pool requires that the oldest, least recently used directory entry in the GBP be reclaimed, provided that the old entry is not associated with a dirty or changed page in the GBP.


Calculating GBP hit ratiosWith the above information in mind, the formula to calculate group buffer pool hit ratio could be:

(SYN.READS(XI)-DATA RETURNED / (SYN.READS(XI)-NO DATA RETURN + SYN.READS(XI)-DATA RETURNED))

SYN.READS(XI)-DATA RETURNED means that a synchronous CF read was issued because a local virtual buffer pool or hiperpool had a page marked as invalid. The page existed in the group buffer pool and was returned.

SYN.READS(XI)-NO DATA RETURN means a synchronous CF read was issued because a local virtual buffer pool or hiperpool had a page marked as invalid and no page was found in the group buffer pool. The recommended hit ratio is >= 70%, provided that the GBP has no problems with directory entries being reclaimed due to cross-invalidations.

Another recommendation is to check periodically for cross-invalidations due to directory entry reclaims.

Automate the issuance of the command every four hours: DIS GBPOOL(*) GDETAIL(INTERVAL) TYPE(GCONN

peoplesoft v8 on zseries using sysplex data sharing haringthis information could include technical...

Documents