impact2014: practical performance troubleshooting

© 2014 IBM Corporation

AAI-2419 Practical PerformanceChris BaileyTodd Kindsfather

Please Note

IBM’s statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.

Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.

The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality. Information about potential future products may not be incorporated into any contract. The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.

Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user’s job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

Introduction to the Speakers

Chris BaileyJava Serviceability and Cloud Integration Architect• 13 years experience developing and deploying Java SDKs

• Recent work focus:‒ Java integration into the cloud‒ Java monitoring, diagnostics and troubleshooting‒ Requirements gathering‒ Highly resilient and scalable deployments

• Contact Details:‒ [email protected]‒ http://www.linkedin.com/in/chrisbaileyibm‒ http://www.slideshare.net/cnbailey/

mailto:[email protected]

http://www.linkedin.com/in/chrisbaileyibm

http://www.slideshare.net/cnbailey/

Introduction to the Speakers

Todd KindsfatherProduct Manager, IBM Application Performance Management• 14 years at Tivoli (now called Cloud & Smarter Infrastructure), with 5 years as a

Services consultant and 9 as a product manager

• Recent work focus:‒ Release of Application Performance Diagnostics on premise and SaaS diagnostics

offering‒ Diagnostics in PaaS environments‒ Requirements gathering and future release planning

• Contact Details:‒ [email protected]‒ http://www.linkedin.com/in/toddkindsfather‒ Twitter: @ToddKindsfather

mailto:[email protected]

http://www.linkedin.com/in/toddkindsfather

5 © 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

Goals of the Talk

■ Introduce a simple general methodology for performance analysis

■ Discuss the common performance bottlenecks

■ Show how to analyse a simple application


Agenda

■ Approaches to performance

■ Layers of the application

■ Identifying and resolving performance issues


Approaches to performance

■ Layered approach– “Bottom up” or “Top down”– Analyze and eliminate layers of the application– Simplify the problem as you go– Ideal for application health check

■ Outside in approach– Start from where performance can be measured– Work along the activity path– Ideal for identified performance problems

■ A hybrid of both approaches can often be useful


Performance baseline

■ Important to have a repeatable and representative performance test

■ Measure baseline performance– Internal measurements affect the performance of what your measuring– External measurements have less impact on system performance

■ Where possible, measure multiple times– Variation will occur between test runs

■ Where possible, ensure consistency– Not just the load test that's run– State of machine and network can have interesting effects


Page response performance benchmark: baseline

PlantsByWebSphereservlet_ShoppingServlet

servlet_ShoppingServlet{2}Shopping{1}

Shopping{4}Shopping_1_1

Shopping_2_2Shopping_2_3


0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

Baseline

Page URL

Avg

Late

ncy

(mic

rose

cond

s)


A Layered Approach

■ Three layers of a deployment:– Infrastructure: Machine hardware and Operating System– Java Runtime: Garbage collection– Java Application: Java application code

■ Each can suffer from resource constraints, typically:– Memory– CPU– Synchronization– I/O

Infrastructure


Infrastructure

■ Typical resource constraints:– Memory: insufficient physical memory results in paging/swapping– CPU: insufficient CPU time limits throughput of the application– I/O: insufficient I/O limits throughput of the application– Synchronization driven by Java runtime/Java application

■ Easy to diagnose■ Easy to resolve (relatively)

■ Note that each can also be caused by deficiencies higher up the stack!


Infrastructure: Memory usage

■ Infrastructure uses memory for:– Backing the process data: OS runtime, Java runtime, Java application– Caching of IO: filesystem and network buffers

■ Lack of physical memory causes:– Reduction and removal of IO caching– Paging/swapping of process memory to disk

■ Paging/swapping is costly for a Java process– Particularly affects Garbage Collection performance

• Paging usually occurs on Least Recently Used basis• All of Java heap is traversed during mark and sweep phases• Least Recently Used does not work well for the Java heap


Infrastructure: CPU usage

■ Insufficient CPU time availability will reduce performance

■ Can occur periodically:– Cron Jobs running batch applications– Database backups

■ Or during periods of high load:– System becomes CPU bound, limiting performance


Detecting infrastructure issues

■ Operating System Tools (eg. “perfmon on Windows)– Paging: using “perfmon” with “Process” counter for “Page Faults/sec”– File Cache: using “perfmon” with “Memory” counter for “System Cache Resident Bytes”– Per process: using “perfmon” with “Process” counter for “% Processor Time”– Per machine: using “perfmon” with “Processor” counter for “% Processor Time”– Network: using “perfmon” with “Network Interface” counter for “Output Queue Length”– Disk: using “perfmon” with “Physical Disk” counter for “Current Disk Queue Length”

■ Health Center

Machine and Process CPU Usage • Visualizes heap usage and gc pause times over time• Identifies memory leaks• Suggests command-line and tuning parameters• Same recommendation logic as GCMV

Native Memory•Detect native memory leaks in application•Determine if external forces are using more memory•Viewcomponents using the most native memory


Paging in perfmon


Resolving infrastructure issues

■ Add more physical resources to the process– Assign more to the: Machine, Guest OS, LPAR, Zone, etc

■ Reduce the physical resource requirements– Reduce the application footprint– Reduce the application CPU usage– Reduce the IO


Page response performance benchmark: Baseline






0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

Baseline

Page URL

Avg

Late

ncy

(mic

rose

cond

s)


Page response performance benchmark: Paging removed






0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

Baseline

Page URL

Avg

Late

ncy

(mic

rose

cond

s)






0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

BaselineNo Paging

Page URL

Avg

Late

ncy

(mic

rose

cond

s)


Effect on page performance

– PlantsByWebSphere 1.7%– servlet_ShoppingServlet{2} 22.8%– servlet_ShoppingServlet 0.3%– Shopping{1} 0.4%– Shopping{4} 21.4%– Shopping_1_1 17.9%– Shopping_2_2 2.8%– Shopping_2_3 16.7%– Shopping_2_4 24.1%– Shopping_2_5 13.8%

■ Improvement in page performance of 0.3% to 24.1%– Biggest gains were for those pages that were already fast

■ Biggest gains are on most performant pages. Total gain is only ~4%

■ Small effect on overall page performance


Garbage Collection Pause Times


Effect on Garbage Collection Pause Times

■ Reduction in:– Maximum pause time: 38%– Average pause time: 13%– Time spent in GC 11%

■ Large effect on GC performance, particularly pause times


Swapping/Paging and Java Applications

■ Swapping/Paging usually occurs on Least Recently Used basis– Live but non-recently used data is written out to disk to free RAM for used data

■ Java however “uses” non-recently used data during GC– In order to determine if the data is “live” and still required by the application.– Any Java heap memory that has been swapped out, must be swapped in during GC

Java Heap

Recently used objectsNon-recently used objects

■ 10 objects are not recently used





Java Heap


Live objects according to GC mark phase

■ 4 objects are not recently used– 6 objects needed to be swapped back in





Java Heap


Live objects according to GC mark phase

■ 0 objects are not recently used– Further 4 objects locations needed to be swapped back in

Java Runtime


Java runtime

■ Typical resource constraints:– Memory: insufficient Java heap results in OutOfMemory or high GC overhead– CPU garbage collection overhead, or driven by Java application– Synchronization driven by Java application– IO driven by Java application

■ Easy to diagnose■ Easy to resolve (relatively)


Detecting Java runtime problems

■ Log and trace analysis:– “Native” heap: OS level logs (ps, svmon, perfmon)– Java heap: verbose:gc output

■ Health Center

Native Memory•Detect native memory leaks in application•Determine if external forces are using more memory•Viewcomponents using the most native memory

Garbage Collection visualization• Visualizes heap usage and gc pause times over time• Identifies memory leaks• Suggests command-line and tuning parameters• Same recommendation logic as GCMV


Too Frequent Garbage Collection

© 2011 IBM Corporation31 JavaOne Session 23401 – The Hidden World of Your Java Application and What its Really Doing

Too Frequent Garbage Collection

Long Garbage Collection Cycles

Garbage collection performanceM

emor

y

Time

Heap Size

Heap Occupancy


Resolving Java runtime problems

■ Add more resources to the Java runtime– Java heap: Increase Java heap size– Native heap: Move to 64bit or reduce Java heap size

■ Reduce the memory requirements– Reduce the Java application footprint


Increased Java heap size


Effect on Garbage Collection Pause Times

■ Reduction in:– Time spent in GC 59%

■ However this is only 4.84% of total time








0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

Baseline

Page URL

Avg

Late

ncy

(mic

rose

cond

s)








0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

Baseline

Page URL

Avg

Late

ncy

(mic

rose

cond

s)






0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

BaselineNo Paging

Page URL

Avg

Late

ncy

(mic

rose

cond

s)


Page response performance benchmark: Java Heap size increased






0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

BaselineNo PagingBigger Heap

Page URL

Avg

Late

ncy

(mic

rose

cond

s)


Page performance improvements

– PlantsByWebSphere 0.4%– servlet_ShoppingServlet 0.0%– servlet_ShoppingServlet{2} 33.2%– Shopping{1} +0.6%– Shopping{4} 4.5%– Shopping_1_1 2.9%– Shopping_2_2 0.1%– Shopping_2_3 2.1%– Shopping_2_4 14.2%– Shopping_2_5 8.2%

■ Improvement in page performance of -0.6% to 33.2%– Without outliers: 0.0% to 14.2%

■ Total gain is only ~4%– Relatively small effect on overall page performance

Java Application


Java application

■ Typical resource constraints:– Memory: insufficient caching affects application throughput and responsiveness– CPU: insufficient threading causes limits on scalability– Synchronsation: synchronized resources limits scalability and throughput of the application– I/O: blocking on I/O limits throughput and responsiveness

■ Hard to diagnose■ Can be expensive (or impossible!) to resolve


Java application CPU usage

■ High CPU usage by Java methods highlight areas of potential optimization– Code is being invoked more than it needs to be

• Easily done with event driven models– An algorithm is not the most efficient

• Easily done if performance is not the focus at development time

■ Fixing CPU bound applications requires knowledge of what code is being run– Identify methods which are suitable for optimisation

• Optimising methods which the application doesn’t spend time in is a waste of your time– Identify methods where more time is being spent that you expect

• “Why is so much of time being spent in this trivial method?”


Java application synchronization

■ Throughput does not increase linearly with load

■ At limit of throughput the CPU is still low– Inability to scale– Not all CPU can be utilized– Limit on throughput and responsiveness

■ Bottleneck where threads need to synchronize with each other for application correctness– Caused by large numbers of threads requiring synchronized resource at the same time– Caused by long hold time by thread that owns resource– Or a mixture of both

© 2012 IBM CorporationPractical Performance: Understand and Improve the Performance of Your Application

1. Performance Overview• Request summary highlights slowest

requests• Plots response times / CPU over time• Breakdown to show hot spots• Distribution chart shows percentage of

requests that are slow

2. Drill down to view Request Instances • Breakdown of response time or CPU for

individual requests

3. Determine the root cause• Select a request instance to view a method

tree that visually identifies bottlenecks• Stack trace is viewable in context• Aggregated method calls also available• Link to application source in IDE

Java Application (using APD Lite)


1

APD Lite Performance Overview


Right click the highlighted row and select “Show Context/Stack Data”

Drill down to Tree View


Right click the highlighted row and select “Show Context/Stack Data” to see SQL call details

Drill down to tree view for more complex case


Right click the highlighted row and select “Show Context/Stack Data”

Drill down to Tree View


Health Center: application method CPU usage


Health Center: application synchronization


ShoppingServlet.deliberateSlowMethod()

private void deliberateSlowMethod() {// ----------------------------------------------------------------// User clicked on the Tulips, let's tip toe through a// slow method// ----------------------------------------------------------------System.out.println("==> STARTING SLOW METHOD");long timestamp = System.currentTimeMillis();long target = timestamp + SLOWTIME;System.out.println("timestamp="+timestamp);System.out.println("resume at="+target);while(timestamp < target) {

timestamp = System.currentTimeMillis();}System.out.println("==> ENDING SLOW METHOD");

}








0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

Baseline

Page URL

Avg

Late

ncy

(mic

rose

cond

s)








0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

Baseline

Page URL

Avg

Late

ncy

(mic

rose

cond

s)






0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

BaselineNo Paging

Page URL

Avg

Late

ncy

(mic

rose

cond

s)


Page response performance benchmark: Java Heap size increased






0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

BaselineNo PagingBigger Heap

Page URL

Avg

Late

ncy

(mic

rose

cond

s)


Page response performance benchmark: deliberateSlowMethod() changed






0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

BaselineNo PagingBigger HeapApp Fix

Page URL

Avg

Late

ncy

(mic

rose

cond

s)


Page performance improvements

■ servlet_ShoppingServlet 80.8% 5x improvement

■ Shopping{1} 94.3% 20x improvement

■ Improvement in page performance of 80 and 95%– 5x and 20x improvement for affected pages

■ Total gain of 48%!


1

APD Lite Performance Overview


Health Center: application method CPU usage


Health Center: application synchronization


Page response performance benchmark: Summary






0

5000

10000

15000

20000

Page Performance

PlantsbyWebSphere

BaselineNo PagingBigger HeapApp Fix

Page URL

Avg

Late

ncy

(mic

rose

cond

s)


Summary

■ Increasing performance of applications is possible using:– Repeatable benchmark– Incremental measurements as changes are made– Repeated testing and verification

■ Infrastructure resources affect performance, but vast majority of performance gains are in the application!

■ Free tools are available to help you see what's going on:– Operating System tools: perfmon, vmstat, ps, svmon, etc– verbose:gc– APD Lite– HealthCenter

■ But you need to use multiple tools to get a complete picture.....

■ Surely there's an easier way?

Java Application


1. APD Lite– Pre-Production only– Collects real time data– Value in 15 minutes– Key features

• On demand collection of code level data in pre-production environments

• Plugs into Eclipse IDE for launch to application source

2. APD Standard – Available as SaaS or on

premises– Suited for all environments– Collects real time and short-

term trend data– Value in 15 minutes– Key features

• Identify app server health at a glance and view KPIs

• Automatic collection of code level data in pre-production or production for diagnosis

• Leverages lightweight infrastructure for seamless monitoring across applications

APD LiteDevelopment Test Production

APD Standard

IBM Application Performance Diagnostics Overview


1. Identify there is a slowdown in response time/throughput for transactions for a JVM over time

2. Identify, Isolate and Diagnose if an application response issue is caused by a WebSphere resource issue:• Heap / GC• Thread Pool• DB Connection Pool etc

3. Diagnose if an application response issue is caused by application code4. Report slow transactions, SQLs, etc. with method tracing turned on dynamically5. Alternative to Traps

Typical User Scenarios


Diagnose if an application response issue is caused by application code

(Slow transaction – Bad SQL etc)


Transaction with High Response Time – All other metrics look ok


Transaction with High Response Time – We should look at the Sequence Data


Problem SQL


Summary

■ Infrastructure resources affect performance– Paging and Garbage Collection much less than you might expect– However, beware of CPU “starvation” from other processes!

■ Vast majority of performance gains are in the application!


References

■ Get Products and Technologies:– IBM Monitoring and Diagnostic Tools for Java:

• https://www.ibm.com/developerworks/java/jdk/tools/

■ Learn:– Health Center InfoCenter:

• http://publib.boulder.ibm.com/infocenter/hctool/v1r0/index.jsp

■ Discuss:– IBM on Troubleshooting Java Applications Blog:

• https://www.ibm.com/developerworks/mydeveloperworks/blogs/troubleshootingjava/– Health Center Forum:

• http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1461– IBM Java Runtimes and SDKs Forum:

• http://www.ibm.com/developerworks/forums/forum.jspa?forumID=367&start=0

https://www.ibm.com/developerworks/java/jdk/tools/

http://publib.boulder.ibm.com/infocenter/hctool/v1r0/index.jsp

https://www.ibm.com/developerworks/mydeveloperworks/blogs/troubleshootingjava/

http://www.ibm.com/developerworks/forums/forum.jspa?forumID=1461

http://www.ibm.com/developerworks/forums/forum.jspa?forumID=367&start=0

Questions?

We Value Your Feedback

Don’t forget to submit your Impact session and speaker feedback! Your feedback is very important to us – we use it to continually improve the conference.

Use the Conference Mobile App or the online Agenda Builder to quickly submit your survey

• Navigate to “Surveys” to see a view of surveys for sessions you’ve attended

71

Thank You

Legal Disclaimer

• © IBM Corporation 2014. All Rights Reserved.• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication,

it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software.

• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results.

• If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete:Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.

• If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete:All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer.

• Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server). Please refer to http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.

• If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete:Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries.

• If you reference Java™ in the text, please mark the first use and include the following; otherwise delete:Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.

• If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete:Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.

• If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete:Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries.

• If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete:UNIX is a registered trademark of The Open Group in the United States and other countries.

• If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete:Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others.

• If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration purposes only.

http://www.ibm.com/legal/copytrade.shtml

http://www.ibm.com/legal/copytrade.shtml

impact2014: practical performance troubleshooting

Technology

application health center

achieve results similar

potential future products

application response issue

gc pause times

high response time

java applications swapping paging

code level data