best practices in batch tuning empowering it staff to reduce cpu and run-time in the enterprise

Best Practices in Batch TuningBest Practices in Batch TuningEmpowering IT Staff to Reduce CPU and Run-Time in the Empowering IT Staff to Reduce CPU and Run-Time in the

EnterpriseEnterprise

Software Engineering of America www.seasoft.com

Agenda

• Did You KnowDid You Know

• Presentation GoalPresentation Goal

• Importance of Batch TuningImportance of Batch Tuning

• Common Problems EncounteredCommon Problems Encountered

• Solving the Problem Solving the Problem

• Remediation OptionsRemediation Options

• SummarySummary


Did You Know!

The Center on Aging & Work / Workplace Flexibility has used U.S. Census data to calculate that an average of 4.6 adults will turn 65 each minute in 2007.

In 2025, an average of 8.0 adults will turn 65 each minute. (McNamara, 2006).

In 2012, there could be as many as 21 million vacant jobs, but just 17 million workers to fill those posts.

Computing Technology Industry Association

Inc.

Flexible schedules 19%

Part-time work options 12%Contracting retired employees 9%Increased compensation 9%

Training or professional 9%development programs

Bonuses for staying on 8%

Job sharing 5%

Delayed retirement plans 4%

Other 4%

No steps taken 60%

Are you taking any steps to retain IT pros who are nearing

traditional retirement age?

Source: ComputerWorld Vital Signs survey of 233 IT professional, May-June 2007. Multiple responses allowed.

Where have all the Baby Boomers gone!


Ford – Launched project Everest to replace mainframe-based Ford Supplier Network procurement system with new ERP-based system. After 4 years and multiple problems, Everest is scrapped. Cost: $400M

McDonald’s - Launched project Innovate to replace multiple custom-built applications (including mainframe accounting systems) with global, ERP-based platform. After a year of R&D and spiraling costs, Innovate is killed. Cost: $170M

Innovate eats McDonalds’ lunchInnovate eats McDonalds’ lunch

Ford fails to climb EverestFord fails to climb Everest

Did You Know! Who Said the Mainframe is DEAD!

http://www.roughtype.com/archives/2006/01/down_the_drain.php


A recent survey* in the UK showed that among 50 mid-to-large size enterprises using mainframes…

• 93% still depend on the mainframe to run mission-critical applications

• 50% say it is too risky to migrate off mainframe

• 28% tried and failed to migrate off the mainframe

• 12% had limited success with a migration off the mainframe

• 8% had tried and succeeded with a migration off the mainframe

*Source: Mainframe still reufusing to die. Survey, Ireland News, 2/28/07

Did You Know! Who Said the Mainframe is DEAD!


Overall Presentation Goal

Explore the key factors that often limit the Explore the key factors that often limit the focus on batch tuning, and discuss the focus on batch tuning, and discuss the technology framework and methods that technology framework and methods that effectively assist performance and capacity effectively assist performance and capacity staff as well as empower other key staff staff as well as empower other key staff members in the identification and resolution members in the identification and resolution of batch tuning opportunities.of batch tuning opportunities.

Discuss methods for identifying the major sources Discuss methods for identifying the major sources of batch inefficiencies that recover the greatest of batch inefficiencies that recover the greatest batch elapsed time, CPU and I/O savings quickly, batch elapsed time, CPU and I/O savings quickly, and with minimum effort.and with minimum effort.


Why Is Batch Tuning Still Important?

• Corporate growth and new business needs have caused Corporate growth and new business needs have caused the introduction of complex applicationsthe introduction of complex applications

• Most Data Centers are operating 24x7 with online Most Data Centers are operating 24x7 with online availability nearly 100%. To accommodate this, the Batch availability nearly 100%. To accommodate this, the Batch window process must be reducedwindow process must be reduced

• More than 50% of critical data processing is done via the More than 50% of critical data processing is done via the batch processbatch process

• Batch processing continues to grow while batch windows Batch processing continues to grow while batch windows continue to shrink continue to shrink


Common Problems of Batch Tuning Practice?

• New forms of z/OS transactions have lead the performance New forms of z/OS transactions have lead the performance analyst to concentrate efforts on WLManalyst to concentrate efforts on WLM

• Lack of efficient JCL coding practices and Standards Lack of efficient JCL coding practices and Standards EnforcementEnforcement

• Serial design of most batch jobs, along with I/O intensity, Serial design of most batch jobs, along with I/O intensity, significantly contributes to the elapsed runtime of batch jobssignificantly contributes to the elapsed runtime of batch jobs



• Existing Performance Analysis Tools:Existing Performance Analysis Tools:– Can be extremely complex to Use Can be extremely complex to Use – Complicate data analysis with a multiplicity of Complicate data analysis with a multiplicity of

reports/options/softwarereports/options/software– Fail to consider batch workload characterization/profilingFail to consider batch workload characterization/profiling– Ignore JCL construct factorsIgnore JCL construct factors

• The Cost of RemediationThe Cost of Remediation

• Batch jobs, although operating correctly, have not Batch jobs, although operating correctly, have not benefited from advanced technology, coding practices or benefited from advanced technology, coding practices or processing methods. processing methods.

•The typical thinking is: The typical thinking is: “If it ain't broke, don't fix it“If it ain't broke, don't fix it”!.”!.


Batch Tuning – Areas to Investigate

OTHERSData not usedExcessive backupsParallelism (Job Split)Redundant Sort Job classes schemeOperator involvement (Automation)Job Priority

DATA IN MEMORY (DIM)DATA IN MEMORY (DIM)VIO

Hiperbatch

BatchPipes

VLF, modules and Execs

Data Base Buffers

Sort Parameters (DFSORT Hipersorting)

VSAM Index and/or Data, BLSR

OPTIMIZE I/OOPTIMIZE I/OBlock sizes (connect time)

Use cache where possible

Level of granularity of serialization (Reduce ENQ time)

Buffering

SYTEM & TECHNOLOGYExploit new addressing scheme

64 bitsWorkload ManagerDASD Technology (ESCON Channels, Stripping)Coupling Facility (JES) TMM

DB2 & IMS Subsystems


Unsystematic Approach

Inaccurate Metrics

Unrepresentative Workload

Inappropriate Evaluation Technique

Ignoring Significant JCL and JOBS Design Factors

Improper presentation of results.

GoalGoal MetricsMetrics SelectSelectWorkloadWorkload AnalyzeAnalyze Imp.Imp.

ChangesChanges ResultsResults

Iteration

1

2

3

4

5

6



GoalGoal MetricsMetrics SelectSelectWorkloadWorkload AnalyzeAnalyze Imp.Imp.

ChangesChanges ResultsResults

IterationIteration

The Challenge

How to Identify specific opportunities by other staff members, e.g. Development, Operations, Production Control

How to minimize data correlation and analysis

How to Model the Batch Timeline and Workload Characterization

How to streamline the remediation phase, minimizing cost and maximizing payoff


Batch Performance Analysis Dimensions

JCL InefficienciesJCL Inefficiencies

Copy SyndromeCopy Syndrome

Deployment CycleDeployment Cycle Target SystemTarget System

System wide focus of performance System wide focus of performance analysisanalysis

Applications complexitiesApplications complexities Performance analysis is a highly Performance analysis is a highly

iterative process and it is difficult to iterative process and it is difficult to implement a practice focused on batch implement a practice focused on batch workloadworkload

Problem data for the batch job Problem data for the batch job population is practically impossible for a population is practically impossible for a specialist to gather and maintain specialist to gather and maintain

Users and application support rely on Users and application support rely on infrastructure for performance studiesinfrastructure for performance studies

Traditional Perf. Analysis

DevelopersDevelopers

Perf. & Capacity Perf. & Capacity Planning AnalystsPlanning Analysts

Sys. ProgrammersSys. Programmers

Production Control Production Control

Batch KnowledgebaseBatch Knowledgebase


A Batch Performance tool should provide:

• Conservative savings in the 15% to 20% range in CPU MIPS and Conservative savings in the 15% to 20% range in CPU MIPS and elapsed runtime. Large organizations should see millions in elapsed runtime. Large organizations should see millions in savings and a return on investment measured in months. savings and a return on investment measured in months.

• Target is Application/System based versus Source code based. Target is Application/System based versus Source code based. This means lower complexity and fast time to value.This means lower complexity and fast time to value.

• Knowledge Base should provide the combined expertise of a Knowledge Base should provide the combined expertise of a Capacity Analyst and a Production Control Analyst.Capacity Analyst and a Production Control Analyst.

• Recommendations should be precise, prioritized, and easy to Recommendations should be precise, prioritized, and easy to understand.understand.


A Batch Performance tool should provide:

• Majority of changes are JCL related and they typically follow the Majority of changes are JCL related and they typically follow the 80/20 rule. 80% or more of the benefits are seen in 20% or less of 80/20 rule. 80% or more of the benefits are seen in 20% or less of the changes.the changes.

• Deliverables should not only facilitate fine tuning of what needs to Deliverables should not only facilitate fine tuning of what needs to run but help identify what can be eliminated. The savings in run but help identify what can be eliminated. The savings in eliminating redundant sorts, or the creation of files that are not eliminating redundant sorts, or the creation of files that are not used, is a 100% efficiency gain for those components eliminated!used, is a 100% efficiency gain for those components eliminated!

• Should not be overhead intensive. Analysis should be based on Should not be overhead intensive. Analysis should be based on history.history.

• The approach should be proactive and embrace a strategy to The approach should be proactive and embrace a strategy to preserve the integrity of the changes. This is much more resource preserve the integrity of the changes. This is much more resource efficient than automating the same change every time an inefficient efficient than automating the same change every time an inefficient component is executed.component is executed.


• Jobs Sorting Already-Sorted Data• Jobs Performing Unnecessary Passes Over Data• Jobs Executing Slow Utilities• Jobs with Inefficient DFDSS/FDR Executions • Jobs with Inefficient DB2 Utility Execution• Jobs Failing During Execution • Unnecessary Passes Over Data

CPU Savings


• Data Created But Not Referenced• Data Sets with Inefficient Block-Size• VSAM Inefficiency Matrix• VSAM Inefficient NSR Buffering• VSAM Candidates for Batch LSR• DFSMS-compressed Data Set Inefficiency Matrix• DFSMShsm Data Set Migration Thrashing • Inefficient Cobol Compile Options

CPU Savings


• Jobs with Allocation Delays• Jobs Waiting for Initiator• Jobs with Excessive Program Run-Time• Jobs Delayed Prior to Init. Selection Phase • Data Set with ENQ Conflicts• Sort Inefficiencies Matrix

Elapsed Time Savings


• Data Sets with Inefficient Block-Size• Over-Allocated DASD Data Sets• Data Created But Not Used• VSAM Inefficiency Matrix

DISK Space Savings


Workload Selection

Workload Selection– Network Flowchart and Workload Selection– Network Flowchart and Critical Path Analysis in a Batch StudyCritical Path Analysis in a Batch Study

A critical Path is that sequence of activities (Jobs) that determine the total elapsed time for a given batch workload to reach a defined point. It identifies which of the activities in a batch workload govern the timing of work during the batch Window.


Workload Selection

Workload Selection– Network Flowchart and Critical Workload Selection– Network Flowchart and Critical Path Analysis in a Batch StudyPath Analysis in a Batch Study

A systematic and reliable study requires a consistent compilation of data from comparable runs. Ex1: Daily Jobs, plot data from a month, (20 samples) Weekly Jobs, plot the data from three months (13 samples) Verify if the end of month run implies extra work

Average job execution times from number of batch cycles Watch Deviation ! Produce exceptional reports after each batch window Trend analysis and monthly reports Modeling the effect of changes. Focus on new CP is path is projected to change Identify Steps that can be moved to another Job (backups, reorgs, etc.)

1. IBM Redbook SYSTEM/390 MVS Parallel Sysplex Batch Performance


Workload Selection

Workload Selection– Network Flowchart and Critical Workload Selection– Network Flowchart and Critical Path Analysis in a Batch StudyPath Analysis in a Batch Study


Workload Selection


Job Analysis Example #1

The problem:

TEMPORARY DATASET CREATED BUT NOT REFERENCED AFTERWARD

Exactly where it occurs:

JOB GZDTP###

PGM=TBR017 STEP= SEL0301 PROCSTEP=

EXEC=000.11.50 CPU=0.02.49 EXCP#=353K

DD=SYSUT2 MODE=OUTPUT ACCESS=QSAM EXCP#=238K

DSN=SYS10020.T235730.RA000.ZMZL0200.R0149276

How to correct it:

DATA IS WRITTEN INTO A TEMPORARY DATASET BUT DOES NOT SEEM TO BE USED AFTERWARDS. THE CREATION OF THE DATA CAN PROBABLY BE ELIMINATED


Job Analysis Example #2

The problem: IDCAMS REPRO EXECUTION FOLLOWED BY COMPLEMENTARY SORTExactly where it occurs: JOB ZMZL0200

PGM=IDCAMS STEP=STEP03 PROCSTEP=EXEC=000.04.01 CPU=0.00.49 EXCP#=62KDD=UNLOAD MODE=OUTPUT ACCESS=QSAM DSN=DM.MPROD.DATA33

EXCP#=31kPGM=ICEMAN STEP=SM03 PROCSTEP=EXEC=000.02.09 CPU=0.00.27 EXCO#=69KDD=SORTING MODE: INPUT ACCESS=BSAMD DSN=DM.MPROD.DATA33

EXCP#=31KHow to correct it:

A SINGLE SORT STEP CAN USUALLY REPLACE THE TWO STEPS OF IDCAMS REPRO AND SORT. IMPLEMENTING THE SINGLE SORT STEP WILL PROVIDE SIGNIFICANT SAVINGS IN ELAPSED TIME, CPU TIME, AND I/O ACTIVITY


What About DB2?

• Improve DB2 Application EfficiencyImprove DB2 Application Efficiency

• Increase DB2 ParallelismIncrease DB2 Parallelism

• Improve Efficiency of DB2 HousekeepingImprove Efficiency of DB2 Housekeeping

• Eliminate DB2 ContentionsEliminate DB2 Contentions


DB2 Analysis Example #1The problem:

A READ-ONLY DB2 THREAD DOES NOT EXPLOIT DB2 QUERY PARALLELISM

Exactly where it occurs:JOB=MDIKIG04 STEP=STEP020 PROCSTEP= PGM=IKJEFT1B

DB2-ID=DBP2 PLAN=MDPPROD TYPE=TSO-BACKGROUNDPACKAGE: VKB_DVP2.COLPROD.MDIKIGSTART-OF-THREAD: 2010/01/18 20.28.09END-OF-THREAD: 2010/01/18 21.16.10

How to correct it:THE APPLICATION PERFORMS EXTENSIVE READ-ONLY OPERATIONS, BUT DB2 HAS NOT USED THE OPTION TO PROCESS THE QUERY IN PARALLEL. PARALLEL DB2 OPERATIONS CAN SIGNIFICANTLY REDUCE THE ELAPSED TIME OF EXTENSIVE QUERIES. IF THIS IS AN IMPORTANT THREAD THAT MUST RUN FASTER, CONSIDER ENABLING DB2 PARALLELISM FOR THIS APPLICATION.


The problem:

A DB2 THREAD HAS ENCOUNTERED INTERNAL LIMITATIONS DURING RID LIST PROCESSING.

Exactly where it occurs:

JOB=MDIKIG04 STEP=STEP020 PROCSTEP= PGM=IKJEFT1B

DB2-ID=DBP2 PLAN=MDPPROD TYPE=TSO-BACKGROUND

PACKAGE: VKB_DVP2.COLPROD.MDIKIG

START-OF-THREAD: 2010/01/12 19.08.02

END-OF-THREAD: 2010/01/13 22.11.10

RID LIST PROCESSING: #SUCCESSES=12 #FAILURES= 237

DB2 Analysis Example #2


How to correct it:

RID LIST PROCESSING CAN HELP REDUCE I/O RESOURCE CONSUMPTION AND ELAPSED TIME. HOWEVER, IF THERE IS NOT ENOUGH RID LIST STORAGE, RESOURCES ARE ACTUALLY WASTED (THE APPLICATION HAS ALREADY CONSUMED RESOURCES AND, UPON FAILURE, MUST REVERT TO A TABLE SPACE SCAN.)

IT IS SUGGESTED THAT THE DB2 ADMINISTRATOR CONSIDER THE FOLLOWING:

1. IF THE PACKAGE WAS BOUND WITH A SMALL AMOUNT OF DATA: THE ACCESS PATH THAT WAS CHOSEN MIGHT BE INEFFICIENT AND MIGHT HAVE CAUSED THE FAILURE. RUNSTATS AND REBIND MAY SOLVE THE PROBLEM.

2. CHANGING THE APPLICATION PATH TO INHIBIT RID LIST PROCESSING (LIST PREFETCH ACCESS PATH).

3. REVIEWING THE APPLICATION LOGIC.

DB2 Analysis Example #2


A PC-based Graphical Analysis Facility (GAF) provides a set of pre-defined Excel spreadsheets of some Global reports

Graphical Analysis Facility (GAF)

JOBHISTJOBHIST SUBGAFSUBGAF

Hlq.. REP%ID%.D%DATE%.T%TIME%Hlq.. REP%ID%.D%DATE%.T%TIME% My Computer (.TXT file)My Computer (.TXT file)

ExcelExcel


GAF - Examples


Remediation Techniques and Options (EXAMPLE)

Success Factors Cost Factors

Reduce MIPS

Consumed Lower

HW/SW Cost


Remediation Techniques and Options

• Implement gradual changes using a systematic approach Implement gradual changes using a systematic approach (minimize the level of risk) (minimize the level of risk)

• Consider JCL refurbishing (< cost)Consider JCL refurbishing (< cost)

• Focus improvement on the Jobs in the Critical PathFocus improvement on the Jobs in the Critical Path

• Fine-tune the base modelFine-tune the base model

• Coordinate the implementation of changesCoordinate the implementation of changes

• Verify results (adjust base line), start againVerify results (adjust base line), start again


Summary

• Tuning the Batch environment can result in significant CPU Tuning the Batch environment can result in significant CPU and elapsed time savingsand elapsed time savings

• The traditional approach to batch tuning makes it difficult to The traditional approach to batch tuning makes it difficult to identify batch application inefficiencies, hinders participation identify batch application inefficiencies, hinders participation by application development teams, and limits the by application development teams, and limits the effectiveness of resultseffectiveness of results

• Identifies and deciphers the batch inefficiencies so that Identifies and deciphers the batch inefficiencies so that application development and support personnel are application development and support personnel are empowered to locate and correct batch issues. empowered to locate and correct batch issues.

Interested in learning more: Schedule an on-site meeting and live demonstration

for your entire IT staff

FOR MORE INFORMATION PLEASE CONTACT: FOR MORE INFORMATION PLEASE CONTACT:

E-mail: [email protected]

Tel: 800.272.7322

Web: www.seasoft.com

mailto:[email protected]

best practices in batch tuning empowering it staff to reduce cpu and run-time in the enterprise

Documents