Introduction There was a time when IT folks could show value by focusing only on operational performance. Today, however, the focus has shifted more to showing the extent to which IT products and services create value and contribute to their organization‟s business objectives. Businesses want to know how IT can help in expanding their company‟s customer base, increasing products and services portfolios, increasing the profit margin, tapping into new markets, identifying activities for outsourcing, and much more. The alignment of IT activities and deliverables to business goals gains even more significance in the world of mainframes, given the high investments associated with them. It is no longer enough to show mainframe performance metrics only in terms of operational efficiency; we need metrics than can reflect the extent to which mainframes help their organization fulfil business goals. That this is not just a fad is very clear from a study undertaken recently by Gartner and Financial Executives International (FEI). According to that study a fairly high number of CIOs are already reporting to CFOs, and the latter have a significant say in all IT investment matters.1
Purpose
The audience of this paper are CIOs, CEOs, CFOs, and other executives whose business operations are heavily dependent upon the performance of mainframes. It examines the main drivers for mainframe performance management by organizations that have decided to continue using them. This paper also highlights key features of the performance management service from HCL.2
Significance of Mainframes
Throughout the history of the mainframe, IBM has retained its dominance,3 though not without its share of controversy.4 It is estimated that IBM‟s mainframe sales will reach $3.3 billion in 2010, which would represent about three-quarters of the overall mainframe market of $4.5 billion.5
For sometime now, there has been a popular belief that mainframes (sometimes referred to as dinosaurs) are living on borrowed time and may not be around in the near future. Writing in 2003, John R. Phelps of Gartner pointed out that a number of developments over the past 15 years6 led to stories about the death of the IBM mainframe, but that reality has been to the contrary. Currently, a great deal of attention is being given to the declining revenue from mainframes and servers in general,7 and once again we are seeing stories about the end of mainframes in the very near future. What is interesting, however, is that IBM has continued to modernize and invest in a range of mainframes, improve pricing, and reach out to new customers, especially in newly industrializing economies,8 which indicates that IBM still has confidence in continued business from mainframes. In fact, IBM is solidifying its position in niche accounts.
2
Mainframe Performance Management - Can Make the Dinosaur Dance
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
niche accounts. For instance, the zEnterprise system (z196)--latest mainframe model from IBM--is targeted at the top 10% to 20% of its customers where the volume of Millions of Instructions Per Second (MIPS) is very high.9 MIPS is one of the most important criterion that IBM uses with its customers when calculating the cost of mainframe license.10
The mainframe story would be incomplete without its customers. Although some customers are on the look out to reduce dependency on mainframes by moving to systems powered by newer technologies, many still continue to use them for historical reasons. Mainframe customers are big companies in data-intensive industries such as distribution, financial services, general business, and public sector that invested in mainframes several years ago. For these customers, reliability, security, and scalability are top priorities,11 and any improvement in the end-to-end performance of mainframes is a very important step in bringing down the cost of owning and running mainframes.
Drivers for Mainframe Performance
In general the main business drivers for mainframe performance improvement include:
Reduce the Total Cost of Ownership (TCO) of mainframe servers
Increase productivity
Make the online applications available to an increasing number of customers without any delay12
Improve customer or end-user experience e.g., comparing auto insurance quotes13 and the like transactions
Increase customer retention14
Eliminate SLA violation
Assuming that IT is a means for achieving business objectives of an enterprise then it is imperative for IT management and staff to be on the same page as the business management and staff in their company. This will not only give all IT activities and decisions a business context but also reveal their significance or the impact on business.
The changing business environment today is compelling IT management to look at ways of obtaining maximum performance from the core mainframe applications and of aligning applications directly to the overall business objectives of their organization. The bulk of the mainframe applications were developed years ago, which have since then undergone enhancements or patchwork. Consequently, they have lost maintainability and traceability to business requirements.
3
Mainframe Performance Management - Can Make the Dinosaur Dance
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Today, more than ever before, there is an ever changing workload being executed on the mainframe. Starting with traditional batch and online workloads, there is an increasing number of transactions running on the mainframe which are invoked from the Web. Additionally, there are queries that are used for complex reporting and analytics in Data Warehouse & Business Intelligence environments.
Poor mainframe performance may be caused by all or a combination of the following problems:
Environment, system and sub-system parameters are not set correctly
Aging and monolithic applications are large and complex, and therefore difficult to integrate with modern systems, which are distributed and agile
Database parameters are not tuned for optimal performance
Applications and databases are poorly designed
Application code and SQLs are not tuned for performance
Online applications may not be available due to the scheduling of jobs in critical path and end-of-day batch jobs
File block size and buffer parameters are not set correctly, leading to high CPU and I/O times
The above-mentioned technical problems need to be addressed by IT teams, given that they lead to the following undesirable consequences:
Applications consuming high volume of MIPS translate into higher licensing cost of mainframes
Poor response time of online applications leads to customer dissatisfaction and decline in customer base
Extended batch windows affects productivity of day-time employees negatively, given that it cuts into regular business hours
Inability of applications to scale up to increasing data volumes leads to SLA violations
Mainframe Performance Services from HCL
This is a comprehensive service from HCL. Using proven methodologies and frameworks, it identifies poorly designed processes, programs, databases, and environment on the mainframe. On the basis of its findings, HCL proposes a solution and remedial strategy. During implementation, HCL enhances the configuration of the mainframe processes, programs, databases, and services such that the overall response and execution time is faster than before. Customers who have availed of this service have been able to reduce MIPS usage for their applications, lower TCO, and utilize the available processing capacity for newer workloads.
4
Mainframe Performance Management - Can Make the Dinosaur Dance
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
HCL Approach As illustrated in Error! Reference source not found., HCL uses its proprietary framework for workload optimization to help customers analyze their workload and identify areas of application tuning. This approach takes into account inputs on the architecture of business applications and databases, the design of online and batch components, and also usage/runtime statistics (from SMF & RMF, performance management tools, history reports, and environment as well system/ subsystem parameters).
Figure 1: HCL proprietary Framework
As highlighted below, the objective of the assessment approach adopted by HCL is to identify poorly performing workloads on the mainframe so that appropriate recommendations could be made and a remedial strategy developed.
5
Mainframe Performance Management - Can Make the Dinosaur Dance
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
PERFORMANCE OPTIMIZATION ASSESSMENT
APPROACH
As shown in Error! Reference source not found., HCL first collects data on various parameters from the client systems and subsystems. It analyzes that data and then proposes a solution, together with a remedial strategy.
Figure 2: Performance Optimization Execution Approach
HCL follows a systematic approach in identifying areas of improvement in mainframe systems and processes. Here is a glimpse of that approach.
In batch jobs, HCL
Examines the critical path in the batch cycle in order to identify undue waiting times
Identifies the most expensive batch jobs, programs, utilities, and recurring job failures
Analyzes performance data reports for selected jobs
In online transactions, HCL
Identifies the most expensive online transactions and recurring failure events
Analyzes performance data reports for selected transactions
For identifying design and architecture problems, HCL
Understands the pain points in the application architecture
Locates the precise problems in the design of database and external interfaces
Analyzes performance data reports for selected jobs and transactions
Data
Collection
Propose
Solutions
Remediate Analysis
Analyze usage
and runtime
data from SMF
and RMF to
identify the
most expensive
workloads
(databases,
subsystems,
batch jobs,
online
transactions,
utilities etc)
Identify areas of
performance
improvement
using
performance
tools, history
reports, and
application
architecture
diagrams
Make recommendations to optimize performance of the workload. Estimate cost, effort and duration required for implementing proposed recommendations aimed at optimizing performance. Prepare Gain Vs Effort matrix to prioritize each recommendation.
Construct high level designs
for each approved
recommendation.
Conduct baseline
performance of the code
(production version) in
controlled test region.
Carry out functional and
performance tests of
impacted components in
controlled test environment.
Compare results from
performance test,
implement agreed upon
recommendations, and
demonstrate benefits.
Provide warranty support
for impacted components.
6
Mainframe Performance Management - Can Make the Dinosaur Dance
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
For identifying specific problems in the database, HCL
Examines contentious issues
Considers or reconsider data compression
Considers if appropriate Indexes are being used by the query
Takes into account the possibility of reorganizing table spaces
Explores if partitioning and segmentation of table spaces would be appropriate
For isolating problems at the system and subsystem levels,
HCL
Analyzes performance data reports for z/OS, CICS, DB2 and IMS systems
Analyzes all system and subsystem parameters
Examines Buffer Pool, Sort Pool, EDM Pool, and DB2 Catalog
PERFORMANCE MANAGEMENT BEST PRACTICES
HCL uses a set of performance management best practices that enable it to:
Identify early signs of performance problems by using performance tools and system related performance reports.
Automate real-time problem detection and resolution by putting in place performance indicators and thresholds.
Prevent the occurrence of problems by using the SMF and RMF data reports to allocate workload resources based on performance. Historically, the stellar reputation earned by mainframes for executing high performance operations continuously has come, in part, from the fact that engineers have been able to allocate system resources proactively. This proactive allocation was not a problem when mainframe systems were siloed, software changes were relatively infrequent, and workloads were predictable.
7
Mainframe Performance Management - Can Make the Dinosaur Dance
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Benefits The overall business benefits that clients have enjoyed from an improvement in the performance of mainframes are lower TCO and increased productivity. However, as illustrated in Table 1 and Table 2, there are other specific advantages as well.
Client: US-based Auto & Home Insurance Company
Area of
Engagement
Performance
Optimization
of applications
in scope
Solution
SQL query
tuning -
overall 60%
gain in
performance
Response
time
improvement
- overall 73%
faster views
New indexes
to improve
performance
Data model
redesign
improvements
WLM goal
mode policy
recommendat
ions -
performance
gain of
around 20%
All small jobs
now run
under a
minute as
compared to
5 – 60
minutes
earlier
Customer Benefits
Reduced TCO
Enhanced cost
effectiveness
Reduced
technology risk
Optimized
business
functions and
their mapping
to system
landscape
Improved
global visibility
of applications
and projects
Reduced time
to take
decisions for
new initiatives
Business
Objectives
Improve the
performance
of the product
in production
and
recommend
improvements
in
consumption,
response time,
database
improvements,
architecture,
WLM, etc
before its
Phase 2
development.
Technology
Mainframe
Technology
(REXX, DB2
V7, ISPF
Services, JCL,
OMEGAMON
for DB2 &
other
technologies)
Engagement
Model/Size
Onsite (30%)
– Offshore
(70%)
Project
Duration
4 months
Table 1: Performance Optimization – US Client
8
Mainframe Performance Management - Can Make the Dinosaur Dance
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Client: German Financial Services Provider with Global
Operations
Area of
Engagement
Performance
Optimization of
applications in
scope
Solution
Expected
Optimization
Addressed
CPU
consumption –
~ 30,000 CPU
Hours
Expected
optimization for
HCLTs
recommendati
ons – Average
~ 15%
Optimization
Recommendations
SQL query
tuning (100+
recommendati
ons)
New and
modified
Indexes to
improve
performance
Application
code
performance
improvements
Architecture &
application
redesign
Customer Benefits
Reduced TCO
Easily
manageable
Define a formal
IT governance
Enhanced cost
effectiveness
Reduced
Technology risk
Optimized
business
functions and
their mapping
to system
landscape
Improved
global visibility
of applications
and projects
Reduced time
to take
decisions for
new initiatives
Business
Objectives
Provide performance optimization recommendations for applications, SQL tuning, related sub system and related areas
Technology
Mainframe
Technology
(COBOL, DB2,
CICS, JCL, DB2
PM, OMEGAMON
for DB2 & other
technologies)
Engagement
Model/Size
Onsite (10%) –
Offshore (90%)
Project Duration
4 months
Table 2: Performance Optimization – German Client
9
Mainframe Performance Management - Can Make the Dinosaur Dance
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
Endnotes
1 One of the 50 questions that the research team asked 482 senior financial executives was who did IT report to in their organization? The response was as follows: 42% reported to the CFO, 33% to the CEO, 16% to the COO, 2% to a chief administrative officer and 7% to other officers. For full details of the survey results and analysis, see John E. Van Decker, “2010 Gartner FEI Technology Study: The CFO as Technology Influencer" (9 April 2010 ID Number: G00175773) http://my.gartner.com/portal/server.pt?open=512&objID=260&mode=2&PageID=3460702&id=1338722&ref=clientFriendlyUrl [October 2010], p. 2.
2 Note: a discussion on the pros and cons of mainframe lies outside the scope of this paper. 3 IBM‟s share of the market for mainframe is believed to be around 90%. Other mainframe vendors include Computer Associates, BEA, Microsoft, Hewlett-Packard, Fujitsu Siemens, Unisys, Sun, and Micro Focus. 4 The latest being antitrust complaints by TurboHercules, a French maker of open-source software for mainframe computers, and T3 Technologies, an American distributor of Flex software that runs mainframes files. On the basis of those complaints, the European Commission has opened an investigation against IBM. For details, see Kevin J. O'Brien, "Europe to Investigate Antitrust Complaints over I.B.M. Mainframes" (July 26, 2010) http://www.nytimes.com/2010/07/27/business/27blue.html [October 2010]. See also "Antitrust: Commission Initiates Formal Investigations against IBM in Two Cases of Suspected Abuse of Dominant Market Position" (July 26, 2010; Ref.: IP/10/1006) http://europa.eu/rapid/pressReleasesAction.do?reference=IP/10/1006&format=HTML&aged=0&language=EN&guiLanguage=en [October 2010]. See also "NEON Files Motion for Partial Summary Judgment in NEON v. IBM" (September 15, 2010) http://www.marketwatch.com/story/neon-files-motion-for-partial-summary-judgment-in-neon-v-ibm-2010-09-15?reflink=MW_news_stmp [October 2010]. Similar concerns have been expressed in a comprehensive study by the Indian Council for Research and International Economic Relations. For details, see Rajat Kathuria, et al., "The Issues of Competition in Mainframe and Associated Services in India" (March 11, 2010) http://openmainframe.org/featured-articles/the-issues-of-competition-in-mainframe-and-associated-servic.html [October 2010]. 5 Kevin J. O'Brien, Ibid. According to Toni Sacconaghi of Bernstein Research, nearly 40% of IBM‟s profits are mainframe-related. Cited in “The Return of the mainframe Back in Fashion” (January 14, 2010) http://www.economist.com/node/15276714?story_id=15276714&source=hptextfeature [October 2010].
10
Mainframe Performance Management - Can Make the Dinosaur Dance
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
6 He is referring in particular to the movement toward client/server platforms and complementary metal-oxide semiconductor (CMOS) technology combined with the availability, reliability, security, scalability, and manageability of mainframes, the growth of the Internet and e-business, and the Y2K problem, all of which led many to believe that the mainframe was living on borrowed time. For details, see John R. Phelps, “The Future of Mainframes Looks Surprisingly Good,” (August/September 2003 issue of zJournal, now a part of mainframezone) http://www.mainframezone.com/it-management/gartner-cio-update-future-of-theibm-mainframe-the-looks-surprisingly-good [October 2010]. 7 According to IDC, the worldwide server revenue in 2009 was said to have declined 18.9% to $43.2 billion when compared to 2008, and worldwide unit shipments declined 18.6% to 6.6 million units over the same period. For details, see IDC, "Worldwide Server Market Rebounds Sharply in Fourth Quarter as Demand for Blades and x86 Systems Leads the Way" (24 Feb 2010) http://www.idc.com/getdoc.jsp?containerId=prUS22224510 [October 2010]. According to one estimate, IBM‟s mainframe revenue took a plunge of 39% in the second quarter of 2009. See Steve Hamm, "IBM Defends Its Big Iron" (August 4, 2009) http://www.businessweek.com/technology/content/aug2009/tc2009084_001429.htm [October 2010]. 8 There are several articles on this subjects. See, for instance, Zacks Investment Research, "IBM‟s Mainframe Wins Customer" (June 30, 2010) http://www.zacks.com/stock/news/36260/Zacks+Analyst+Blog+Highlights%3A+Union+Pacific%2C+IBM%2C+Hewlett+Packard%2C+Vornado+Realty+and+KKR+Financial [October 2010]. See also Robert L. Mitchell, ""Morphing the mainframe" (February 6, 2006) http://features.techworld.com/operating-systems/2229/morphing-the-mainframe/ [October 2010]. 9The zEnterprise systems is said to start at about $1 million and according to Lou Miscioscia, an analyst with Collins Stewart, they are expected to generate a profit margin of about 70 percent, vs. a 46 percent margin for the company as a whole. Cited in By Katie Hoffman, "IBM Mainframes: Boring but Profitable" (July 22, 2010) http://www.businessweek.com/magazine/content/10_31/b4189041885991.htm [October 2010]. See also IBM, "IBM zEnterprise System," http://www-03.ibm.com/systems/z/hardware/zenterprise/ and also "Success Stories," http://www-03.ibm.com/systems/z/success/index.html [October 2010]. According to Martin Kennedy, the managing director of Citigroup's enterprise systems infrastructure, "we will be able to collapse multiple existing large systems into the new water-cooled z196, which is a huge step in the company's attempt to shift more of its IT dollars away from internal operations and maintenance and toward customer-facing efforts. For more details, see Bob Evans, "Global CIO: IBM's Blazing New Mainframe Wins Raves from Citigroup," (September 2, 2010) http://www.informationweek.com/news/global-cio/interviews/showArticle.jhtml?articleID=227200199&pgno=2&queryText=&isPrev= [October 2010]. See also John R. Phelps and
11
Mainframe Performance Management - Can Make the Dinosaur Dance
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
And Mike Chuba, "IBM Adds Integrated Management Role for Mainframe," July 27, 2010; ID:G00205553) http://my.gartner.com/portal/server.pt?open=512&objID=260&mode=2&PageID=3460702&resId=1411013&ref=QuickSearch&sthkw=mainframe [October 2010]. 10 MIPS reflect the mainframe‟s footprint, and over the past decade it has witnessed a steady rise with installed capacity reaching a high of 11.1 million in 2007. For details, see Timothy Prickett Morgan, "The IBM Mainframe Base: Alive and Kicking" (July 10, 2007) http://www.itjungle.com/big/big071007-story01.html [October 2007]. 11 For a quick overview on this aspect, see IBM, "The IBM Enterprise System Advantage - Running the World's Most Sophisticated Business Transactions," http://www-03.ibm.com/systems/migratetoibm/getthefacts/market.html [October 2010]. See also Anne Rawland Gabriel, "Debate over the Future of Mainframe Computing Rages On" (September 16, 2008) http://www.wallstreetandtech.com/it-infrastructure/210601591?pgno=1 [October 2010]. 12 For instance, in the future banks need to accommodate for the increasing number of customers accessing mobile banking application. According to a study by Berg Insight, "the number of active users of mobile banking and related financial services worldwide is forecasted to increase from 20 million in 2008 to 913 million in 2014." Berg Insight, "Mobile Banking and Payments," p. 3, http://www.berginsight.com/ReportPDF/ProductSheet/bi-mbp-ps.pdf [September 2010]. Anticipating such high volume, IBM has developed a 5.2GHz chip - considered to be the highest speed rating to date - for its fastest mainframe computers. See Brooke Crothers, "IBM Ships 5.2GHz Chip, Its Fastest Yet" September 1, 2010) http://news.cnet.com/8301-13924_3-20015297-64.html#ixzz10uOqdeX8 [September 2010]. 13 Ryan Arsenault‟s observation captures the views of many experts thus: “[t]he problem is not the mainframe platform, but the fact that many insurance companies are making use of legacy mainframe application code which was originally designed to provide quotations to real people.” What is needed is an improvement in the performance of hardware and software applications that use computer resources for running a transaction such as generating insurance quotes. For details, see Ryan Arsenault, "Mainframes Run into Performance Problems with Online Insurance Comparison Shopping," http://itknowledgeexchange.techtarget.com/mainframe-blog/mainframes-run-into-performance-problems-with-online-insurance-comparison-shopping/ [September 2010]. 14 Improving performance of business applications is a key to delivering business value to customers and their end-users. A recent survey by the Aberdeen Group found that companies that monitored and measured application performance at the point of consumption by end users had few end user complaints. For full details of the survey, see Jeffrey Hill, “End-User Experience Monitoring and Management” [August 2010]. http://v1.aberdeen.com/launch/report/benchmark/6778-RA-end-user-experience-monitoring.asp [October 2010]. Writing about the excessive significance given to operational performance metrics in the
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
12
Mainframe Performance Management - Can Make the Dinosaur Dance
post 1990s era, Michael Bitterman noted that "IT continues to focus on operations and...explain „how the clock works‟ when users are asking, „what time is it?‟ The availability of tools for monitoring hardware and network performance [has given] IT the ability to overwhelm users with statistics regarding IT performance. Users [become] baffled by the metrics that show that mainframe availability was 99.99% yet they couldn‟t enter data or access systems. For the complete report, see Michael Bitterman "IT Metrics for the Information Age" (July 25, 2004) http://www.performance-measurement.net/news-detail.asp?nID=198 [October 2010].
AUTHORS
Srinivas Murthy Potharaju, ETS Mainframe CoE HCL Technologies, Hyderabad Dr Usha Thakur, ATS Technical Research HCL Technologies, Chennai
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
13
Mainframe Performance Management - Can Make the Dinosaur Dance
ABOUT HCL
HCL Technologies
HCL Technologies is a leading global IT services company, working with clients in the areas that impact and redefine the core of their businesses. Since its inception into the global landscape after its IPO in 1999, HCL focuses on „transformational outsourcing‟, underlined by innovation and value creation, and offers an integrated portfolio of services including software led IT solutions, remote infrastructure management, engineering and R&D services and BPO. HCL leverages its extensive global offshore infrastructure and network of offices in 31 countries to provide holistic, multi-service delivery in key industry verticals including Financial Services, Manufacturing, Consumer Services, Public Services and Healthcare. HCL takes pride in its philosophy of „Employees First‟ which empowers our 72,267 transformers to create real value for customers. HCL Technologies, along with its subsidiaries, had consolidated revenues of US$ 3.1 billion (Rs. 14,101 crores), as on 31st December 2010
About HCL Enterprise
HCL is a $5.7 billion leading global technology and IT enterprise comprising two companies listed in India - HCL Technologies and HCL Infosystems. Founded in 1976, HCL is one of India's original IT garage start-ups. A pioneer of modern computing, HCL is a global transformational enterprise today. Its range of offerings includes product engineering, custom & package applications, BPO, IT infrastructure services, IT hardware, systems integration, and distribution of information and communications technology (ICT) products across a wide range of focused industry verticals. The HCL team consists of over 79,000 professionals of diverse nationalities, who operate from 31 countries including over 500 points of presence in India. HCL has partnerships with several leading Global 1000 firms, including leading IT and Technology firms. For more information, please visit www.hcl.com
© 2011, HCL Technologies. Reproduction Prohibited. This document is protected under Copyright by the Author, all rights reserved.
14
Mainframe Performance Management - Can Make the Dinosaur Dance