optimizing z/os batch

Optimizing z/OS Batch

Large Systems Update 2011

Martin PackerIBM Worldwide Banking Center of Excellence

[email protected]

© 2011 IBM Corporation2

NoticesThis information was developed for products and services offered in the U.S.A.

Note to U.S. Government Users Restricted Rights — Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.

IBM may not offer the products, services, or features discussed in this document in other countries. Consult your local IBM representative for information on the products and services currently available in your area. Any reference to an IBM product, program, or service is not intended to state or imply that only that IBM product, program, or service may be used. Any functionally equivalent product, program, or service that does not infringe any IBM intellectual property right may be used instead. However, it is the user's responsibility to evaluate and verify the operation of any non-IBM product, program, or service.

IBM may have patents or pending patent applications covering subject matter described in this document. The furnishing of this document does not give you any license to these patents. You can send license inquiries, in writing, to: IBM Director of Licensing, IBM Corporation, North Castle Drive Armonk, NY 10504-1785 U.S.A.

The following paragraph does not apply to the United Kingdom or any other country where such provisions are inconsistent with local law: INTERNATIONAL BUSINESS MACHINES CORPORATION PROVIDES THIS PUBLICATION "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF NON-INFRINGEMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Some states do not allow disclaimer of express or implied warranties in certain transactions, therefore, this statement may not apply to you.

This information could include technical inaccuracies or typographical errors. Changes are periodically made to the information herein; these changes will be incorporated in new editions of the publication. IBM may make improvements and/or changes in the product(s) and/or the program(s) described in this publication at any time without notice.

Any references in this information to non-IBM Web sites are provided for convenience only and do not in any manner serve as an endorsement of those Web sites. The materials at those Web sites are not part of the materials for this IBM product and use of those Web sites is at your own risk.

IBM may use or distribute any of the information you supply in any way it believes appropriate without incurring any obligation to you.

Information concerning non-IBM products was obtained from the suppliers of those products, their published announcements or other publicly available sources. IBM has not tested those products and cannot confirm the accuracy of performance, compatibility or any other claims related to non-IBM products. Questions on the capabilities of non-IBM products should be addressed to the suppliers of those products.

This information contains examples of data and reports used in daily business operations. To illustrate them as completely as possible, the examples include the names of individuals, companies, brands, and products. All of these names are fictitious and any similarity to the names and addresses used by an actual business enterprise is entirely coincidental.

COPYRIGHT LICENSE:

This information contains sample application programs in source language, which illustrates programming techniques on various operating platforms. You may copy, modify, and distribute these sample programs in any form without payment to IBM, for the purposes of developing, using, marketing or distributing application programs conforming to the application programming interface for the operating platform for which the sample programs are written. These examples have not been thoroughly tested under all conditions. IBM, therefore, cannot guarantee or imply reliability, serviceability, or function of these programs. You may copy, modify, and distribute these sample programs in any form without payment to IBM for the purposes of developing, using, marketing, or distributing application programs conforming to IBM's application programming interfaces.


Trademarks

This presentation contains trade-marked IBM products and technologies. Refer to the following Web site:

http://www.ibm.com/legal/copytrade.shtml

http://www.ibm.com/legal/copytrade.shtml

© 2011 IBM Corporation

Abstract

Batch performance optimization remains a hot topic for many customers, whether merging workloads, supporting growth, removing cost or extending the online day.

This presentation outlines a structured methodology for optimizing the batch window, incorporating techniques written about in a Redbook written by experts from around the world. This methodology is well-structured and draws on information every installation should have access to.


Agenda

About The Redbook

Context

Analysis Methodology

Analysis Steps

Governance

Antipatterns

Tools


Agenda

About The Redbook Context

Analysis Methodology and Steps

Governance

Antipatterns

Tools


About The Redbook

Written by a worldwide team in Summer 2011:–Alex Louwe-Kooijmans (ITSO Poughkeepsie)–Jan van Cappelle (IBM Netherlands)–Martin Packer (IBM United Kingdom)–Tomohiko Kaneki (IBM Japan)

Follows on from 2009 Redbook “Batch Modernization on z/OS” SG24-7779

–Tries hard not to repeat its material

Meanwhile in the other room–Other residents wrote about modern batch containers–They were “beyond the firewall”


Agenda

About The Redbook

Context Analysis Methodology and Steps

Governance

Antipatterns

Tools


Problem Statements

Context is about turning an initial problem statement (or question) into a better one.

Initial problem statements look like:1)“Between 10:00 am and 4:00 pm we had bad online transaction response

times.”2)“The time window to run my batch is too short.”3)“The processor utilization is too high.”

Initial questions look like:1)“When we merge company A with company B will there still be enough time to

run our batch jobs?”2) “Our batch schedule is so complex we don’t know how it works anymore. We

are worried that when there are problems we won’t be to figure out how it works.”

3) “Can you help us reduce our batch processing costs by 20%?”4) “There is no specific department responsible for our batch schedule, every

department has it own standards and, no one thinks about the overall corporate perspective and requirements.”

5) “We want to clean up our batch schedule but do not know where to start. Can you help?”


Better Problem Statements

We think the following are better versions of the 3 problem statements you just saw:

1)“Between 10:00am and 4:00pm online transaction response times exceeded the service level agreement by 50%. This is causing 100 user complaints a day. We have identified that batch running in the online day is causing a processor capacity shortage.”

2)“We have between 8:00pm and 6:00am to run our batch in. In particular we have to get the replenishment application completed by 3:15am. Today we finish around 2:30am except for when we have a problem...”

3)“Batch is driving our processor capacity requirements (or our software bill because it drives the peak of our Rolling 4 Hour Average). We need to reduce this peak by 25% to help manage our costs.”


Roles

We may have an “idealised” view of roles– Both on the customer and IBMer “side”


Roles ...

Translating initial problem statements into better ones takes the right people– Initial statements tend to come from the business

• Client business expert talks to IBM representative• Sometimes, though, IT acts proactively

– IBM Batch Subject-Matter-Expert (SME) needs to help improve the problem statement– SME also needs to shape the project

• And maybe assemble a team with other technical SME's in


A Good Business Case

Because Batch Optimization projects are extensive a good business case is essential

Some drivers include:

–Processor usage is too high and a reduction in processor usage lowers costs.

•The reduction of million service units (MSU) lowers costs.

–When batch processing has bottlenecks it can impact the business clients and there is a risk of not meeting Service Level Agreements (SLAs).

•When SLAs are not met, it impacts the business clients and you might have to pay a penalty for not meeting your SLA. This represents a risk of profit loss.

–The client wants to simplify the batch schedule.

•The advantages are lower maintenance costs, fewer skilled operators are needed, and there is less risk in your batch schedule.

–The client wants to reduce people time by cleaning up the batch schedule.

•When your batch schedule is cleaned up you need less time to maintain the schedule and less time to develop additional function, because the schedule is now easier to understand.

–The client wants to reduce hardware resources.

•Reducing your hardware resources such as disk space and network usage lowers your costs.

–The client wants to reduce errors in their batch schedule.

•When your schedule is cleaned up and simplified the probability of errors is reduced and it is easier to meet the SLA.


But Consider The Costs, Too

Solutions can have financial impacts, introduce risk, or require a lot of effort. For example:

– The removal of 30,000 obsolete jobs from a schedule of 300,000 jobs. When removing obsolete jobs you also have to review and correct any dependencies.

– The removal or changes to 10% incorrect dependencies in a schedule of 300,000 jobs.

– The installation of BatchPipes/MVS to create more parallelism. But this requires you to acquire and install this program product.

– Required hardware upgrades in order to meet the SLA.

A good business case would, of course, have far more benefits than costs


Agenda

About The Redbook

Context

Analysis Methodology and Steps Governance

Antipatterns

Tools


Analysis Methodology

At this level the methodology applies equally well to batch on all platforms

Comprises 3 steps:

1) Initialization• Standard Project Management “kick off”• We emphasise it because of the complexity

2) Analysis• The interesting part :-)

3) Implementation• Again, similar to “standard” Project Management• Focus on implementing and measuring• Emphasised because of the complexity


Analysis


Setting The Strategy

An important first step– Sets the technical direction based on

• System conditions• Proclivities of the Batch workload

Woven from 7 strategies:– Ensuring the system is properly configured– Implementing Data In Memory (DIM)– Optimizing I/O– Increasing parallelism– Reducing the impact of failures– Increasing operational effectiveness– Improving application efficiency

These strategies described in more detail in:– SG24-2557 “Parallel Sysplex Batch Performance” – SG24-7779 “Batch Modernization on z/OS”


Setting The Strategy – In The z/OS Environment

In the book we take platform-neutral “7 strategies” and make them more z/OS specific

Some of the strategies can have resource implications:- We advocate the use of RMF and other SMF to understand the resource picture

Some require trawling through eg Operations logs- Especially if you want to know when certain events occur

Application efficiency (Strategy 7) is especially difficult to assess- Best left to “Understanding the batch landscape”


7 Strategies – Some z/OS Examples

1) Ensuring the system is properly configured: Ensure the WLM setup is appropriate for batch

2) Implementing Data In Memory: Using VSAM LSR buffering adequately

3) Optimizing I/O: Install enough disk I/O bandwidth

4) Increasing Parallelism: Use BatchPipes/MVS

5) Reducing Impact Of Failures: Use DFSORT / ICETOOL capabilities to “cleanse” data

6) Increasing Operational Effectiveness: Use Tivoli Workload Scheduler to automate batch

7) Application Efficiency: DB2 for z/OS SQL Tuning


Understanding The Batch Landscape (or “Batchscape”)

Complexity is the biggest obstacle to managing batch– Tens or hundreds of thousands of jobs– Many hundreds of thousands of data sets and database objects– etc...

Understanding the batch landscape is important – and difficult

Try to understand:– Naming conventions – and their degree of observance – Job functions – for the important jobs– Application ownership boundaries– Users of database objects– Critical path and end points– Reliability– Job times

… amongst other elements of the batch landscape

You probably already do understand the systems and database managers and networks...

To prepare for the next stage you need to derive a short list of jobs to examine in detail– Probably no more than 30 to start with– Easily extensible later– Whether “critical” or “typical” they have to be worth tuning


Application Boundaries


Understanding The Batch Landscape – In The z/OS Environment

Main sources of information are:- Tivoli Workload Scheduler

• Application Description database- SMF Type 30 Job- and Step-End Records

• Give names, runtime and CPU information• Useful for characterising jobs

- SMF Type 101 Accounting Trace• Correlation ID is usually Job Name• For IMS it's more complicated than just Correlation ID• Connection Type (QWHCATYP) can be used to discard non-batch

records• Also gives very good time breakdown below step level

There's lots of other information which helps build the total Batch Landscape view- System-Level- Subsystem-Level- WLM Service Class Level


Job Naming Conventions


Selecting The Jobs To Tune

Jobs for detailed study might be chosen for a variety of reasons:– CPU Intensive– Long Running– On “Critical Path”– Problematic– Typical of their type– In important applications

Can only realistically look at a few tens of jobs in depth– But these are the ones that are really important to tune

Be prepared to add additional jobs to your initial list


Deriving Detailed Tuning Actions

Remember detailed tuning actions have to fit into the overall strategy

– But without these actions there is no progress

When considering a system there are many resources to tune or provision

– Yes, System-Level counts as a “Detailed” tuning action

Likewise database managers

Jobs are the obvious things to tune:– Speeding up individual steps– Breaking up jobs for parallelism– Removing job dependencies

• Especially spurious, scheduler-induced ones


Speeding Up Individual Steps

Generally easier and lower risk than the 2 “break up” scenarios

Long-running steps are readily identifiable. – Rough guide: Steps > 10% of job’s run time worth considering

Examples of how you might tune a step’s run time are:– Tune the application code

• Might also reduce processor consumption– Tune data set access– Split the step into clones

• running against a subset of the data– Tune DB2 aspects

• For example a locking problem• Use DB2 Accounting Trace (SMF 101)

Many steps comprise more than one phase. Most well-known example is a sorting step.

– A sorting step has three phases:1) Input phase2) Work phase3) Output phase


Breaking Up Jobs For Parallelism / Removing Job Dependencies

Though the mechanics are different the analysis is similar

Reasons why one step runs after another include:– One step writes a data set another step needs to

process– One step accesses DB2 in a way that requires another

step to be scheduled later– Someone added function to a pre-existing job by

simply adding steps– The original design didn’t consider whether steps were

independent or dependent on each other

As you can see the word “step” could be replaced by “job” in the above examples

– Or even “step phase”


Dependency Analysis

Dependencies are either:– In the schedule (Job-level)– Step-serialised (Step-level)

For data sets use Life Of A Data Set (LOADS) analysis:– Some patterns suggest a dependency:

• A writer followed by a reader• A writer followed by a writer• A reader followed by a reader• A reader followed by a writer

For DB2 etc it's much more difficult


Agenda

About The Redbook

Context


Governance Antipatterns

Tools


Governance

We define governance as:– “... the formal process and policies for controlling, directing,

and making batch decisions”

We go on to say:“Governance empowers you with consistent management,

cohesive policies, guidance, and decision rights for your batch actions and solutions. You can use these processes to verify your batch performance.”

An important part of it is creating the Batch Design Authority (BDA)

To create governance:1) Appoint the Batch Design Authority.2) Revise the standards and guidelines.3) Communicate the standards and guidelines.4) Set up a baseline and monitor.5) Start making improvements.6) Control and perform quality assurance.


Agenda

About The Redbook

Context


Governance

Antipatterns Tools


Antipatterns

Not to be confused with “sins”– Difference depends on whether the “sin” matters

Pattern:– A named solution to a recurring problem in a

contexts

Anti-pattern:– Similar to a pattern but leading to negative

consequences

Of course it's not always clear which it is– In the book we have at least one example of this


Antipatterns

The antipatterns we wrote about in the book are:– Incorrect usage of a special resource– Incorrect usage of a start time– Invalid relationship– Obsolete dummy jobs or dummy operations– Actual starttime not equal to endtime last predecessor– Workload Manager unhelpfully set up for batch– Redundant copies of files– Redundant image copies– Batch prevented from effectively using sysplex-wide

resources

We could've written about many more, given an infinite number of monkeys


Agenda

About The Redbook

Context


Governance

Antipatterns

Tools


Tools

We wrote about– Tivoli Workload Scheduler for z/OS (TWS) tools– z/OS tools– Batch Schedule Performance Optimization (BSPO)

Particularly for the z/OS tools we sought to avoid duplicating what was in SG24-7779


Tivoli Workload Scheduler Tools

Operational Efficiency:– Query Current Plan (QCP) and Modify Current Plan (MCP) show

current Production status– Manage Parallelism by defining parallel servers on the “workstation”

Monitoring:– Job errors– Jobs delayed beyond latest start time– Jobs running unexpectedly long

Forecasting:– Long Term Plan Current Plan

Critical Path Analysis

Automatic Job Restart

Promoting to a “higher” WLM Service Class

...


An Example of TWS Graphical Support


z/OS Tools - SMF


z/OS Tools – SMF ...


Batch Schedule Performance Optimization

When a customer complains that:–The Batch Schedule does not fit any more in the time line–Critical endpoints are not met –The batch schedule is to complex

Then it is time to visit the client and analyze what is wrong

After the first quick analyses we can advise performing some steps of the seven step strategy or we advise to bring the batch schedule under governance and after that optimize the batch schedule using BSPO


Batch Schedule Performance Optimization

BSPO is:

– You have your batch schedule under governance

– You start optimizing your batch schedule using the anti-pattern versus pattern strategy.

– An Anti-pattern is a do not.

• e.g. do not use superfluous relationships.

• A pattern is how things have to be.

– e.g. only use those relationships you really need.

• You can use Source2VALUE™ tooling to find the anti patterns you have to change in a pattern.

– The Source2VALUE™ tooling delivers lists with anti patterns.

• When you have the list with anti-patterns you start to change them in the to be situation.

– Source2VALUE™ is a product of Omnext BV

• Has an agreement with IBM to work together on BSPO.


More information on zEnterprise

IBM zEnterprise landing page: http://www.ibm.com/systems/z/hardware/zenterprise/index.html

IBM zEnterprise 114 (z114): http://www.ibm.com/systems/z/hardware/zenterprise/z114.html

IBM zEnterprise Events Landing Page: http://www.ibm.com/systems/breakthrough

IBM software for zEnterprise: http://www.ibm.com/software/os/systemz/announcements

IBM System Storage: http://www.ibm.com/systems/storage/product/z.html

IBM Global Financing: http://www.ibm.com/financing/us/lifecycle/acquire/zenterprise/

IBM Services for zEnterprise: http://www.ibm.com/services/us/gts/zenterprise/index.html

IBM zEnterprise / System z Redbooks Portal: http://www.redbooks.ibm.com/portals/systemz

http://www-03.ibm.com/systems/z/hardware/zenterprise/index.html

http://www-03.ibm.com/systems/z/hardware/zenterprise/index.html

http://www.ibm.com/systems/z/hardware/zenterprise/z114.html

http://www.ibm.com/systems/breakthrough



http://www.ibm.com/software/os/systemz/announcements

http://www.ibm.com/software/os/systemz/announcements

optimizing z/os batch

Technology