javaone 2014: java debugging
DESCRIPTION
TRANSCRIPT
© 2014 IBM Corporation
Chris Bailey – IBM Runtime Monitoring and Diagnostics
30th September 2014
Java DebuggingTools and Tricks for Troubleshooting
Document number
© 2014 IBM Corporation
Important Disclaimers
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED.
ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES.
ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE.
IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE.
IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF:
- CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS
2
© 2014 IBM Corporation3
Introduction to the speaker Chris Bailey
IBM Runtime Monitoring and Diagnostics Architect- 14 years working with Java and JVM technologies
Recent work focus:- Java monitoring, diagnostics and troubleshooting- Java integration into the cloud- JavaScript monitoring, diagnostics and troubleshooting
My contact information:- [email protected] http://www.linkedin.com/in/chrisbaileyibm- http://www.slideshare.net/cnbailey/- @Chris__Bailey
© 2014 IBM Corporation4
OutOfMemory
© 2014 IBM Corporation5
Java Memory Usage Maximum memory available to a process is determined by architecture
– 32bit process is limited to 2^32 bytes = 4GB– 64bit process is limited to 2^64 bytes = 16 EiB
● Example for 32bit:
Amount and location of memory used for “OS and C Runtime” determined by OS
0 GB 4 GB
0x0 0xFFFFFFFF0x800000000x40000000 0xC0000000
OS and C-Runtime Java Heap(s)
-Xmx
Native Heap
© 2014 IBM Corporation6
Layout by OS Windows 32bit:
Windows 32bit with /3GB:
0 GB 4 GB
0x0 0xFFFFFFFF0x800000000x40000000 0xC0000000
OS and C-RuntimeJava Heap(s)
-Xmx
Native Heap
Libraries
0 GB 4 GB
0x0 0xFFFFFFFF0x800000000x40000000 0xC0000000
OS and C-RuntimeJava Heap(s)
-Xmx
Libraries
Native Heap
© 2014 IBM Corporation7
Layout by OS Linux 32bit:
Linux with hugemem kernel:
0 GB 4 GB
0x0 0xFFFFFFFF0x800000000x40000000 0xC0000000
KernelJava Heap(s)
-Xmx
Native Heap
0 GB 4 GB
0x0 0xFFFFFFFF0x800000000x40000000 0xC0000000
KJava Heap(s)
-Xmx
Native Heap
© 2014 IBM Corporation8
Native Heap Usage
Java Heap Native Heap
© 2014 IBM Corporation9
Native Heap Usage
Java Heap Native Heap
© 2014 IBM Corporation10
OutOfMemoryErrors An OutOfMemoryError occurs if the Java heap or Native heap is exhausted
– Or the PermArea (for HotSpot before Java 8)
Tools are needed to track the memory usage of each of the pools:– Kernel/OS: OS (malloc) managed– Native Heap: OS (malloc) managed– Java Heap: Garbage Collection managed
Malloc managed memory needs to be tracked at the OS level*
Garbage Collection needs to be tracked at the Java runtime level
* The JVM can grab this information from the OS for you
© 2014 IBM Corporation11
OS Level Memory Monitoring
#!/bin/shPID=$1INTERVAL=3
# Echo time at start of monitoring.echo timestamp = `date +%s`
# Echo the interval frequency.echo "ps interval = $INTERVAL"
# Run the system command at intervals.while ([ -d /proc/$PID ]) do ps -p $PID -o pid,vsz,rss sleep $INTERVALdone
Linux: “ps” utility Windows: “perfmon” Virtual Bytes counters
Can be collected to CSV file
© 2014 IBM Corporation12
OS Level Memory Monitoring
#!/bin/shPID=$1INTERVAL=3
# Echo time at start of monitoring.echo timestamp = `date +%s`
# Echo the interval frequency.echo "ps interval = $INTERVAL"
# Run the system command at intervals.while ([ -d /proc/$PID ]) do ps -p $PID -o pid,vsz,rss sleep $INTERVALdone
Linux: “ps” utility Windows: “perfmon” Virtual Bytes counters
Can be collected to CSV file
© 2014 IBM Corporation13
Visualization of OS level memory monitoring● Garbage Collection and Memory Visualizer (GCMV):
- Available from
● Tool to analyze Java Heap and “Native” Heap memory
● Provides Graphical Display of Data- Allows zooming and cropping- Allows change of axes value and units- Allows comparison of multiple files
● Analysis and Recommendations- Statistical summary and recommendations- Flags errors and warnings
● Supports Linux, Windows, AIX, i5/OS and z/OS
© 2014 IBM Corporation14
Runtime Monitoring using JVM Tools IBM Health Center (IBM JVMs*)
– -Xhealthcenter
Machine and process memory usage
Native heap breakdown
* limited support exists for HotSpot
Oracle Mission Control (HotSpot JVMs)– -XX:+UnlockCommercialFeatures – -XX:+FlightRecorder
Machine and process memory usage
© 2014 IBM Corporation15
OS Level Memory Profiling
Windows: UMDH
Takes snapshot of the current native heap
Tracks allocating stack traces
Second snapshot is taken and stack traces for outstanding memory is produced
Detailed on Microsoft website:
http://support.microsoft.com/kb/268343
Also check out “debugdiag”
000000AD bytes in 0x1 allocations (@ 0x00000031 + 0x0000001F) by: BackTrace00468 ntdll!RtlpNtMakeTemporaryKey+000074D0 ntdll!RtlInitializeSListHead+00010D08 ntdll!wcsncat+00000224 leakyjniapp!Java_com_ibm_jtc_demos_LeakyJNIApp_nativeMethod
Linux:Debug Malloc / Malloc Interception
Approach 1:– Intercept calls to malloc and free– Create a stacks allocation table– IBM provides a Linux Memory Tracker
Approach 2: valgrind
valgrind --trace-children=yes --leak-check=full
LEAK SUMMARY: definitely lost: 63,957 bytes in 69 blocks. indirectly lost: 2,168 bytes in 12 blocks. possibly lost: 8,600 bytes in 11 blocks. still reachable: 5,156,340 bytes in 980 blocks. suppressed: 0 bytes in 0 blocks.Reachable blocks (those to which a pointer was found) are not shown.
© 2014 IBM Corporation16
Garbage Collection Monitoring IBM JVMs:
– -verbose:gc – -Xverbosegclog:<name>,X,Y
Visualization available in GCMV for both IBM and Oracle verbose:gc
Oracle JVMs:– -XX:+PrintGCDateStamps– -XX:+PrintGCDetails– -XX:+PrintGCTimeStamps– -Xloggc:<name>
© 2014 IBM Corporation17
Runtime GC Monitoring IBM Health Center (IBM JVMs*)
– -Xhealthcenter
Used Heap and Pause Times
Allocation Profiling
* limited support exists for HotSpot
Oracle Mission Control (HotSpot JVMs)– -XX:+UnlockCommercialFeatures – -XX:+FlightRecorder
Used Heap and Pause Times
Allocation profiling
© 2014 IBM Corporation18
Java Heap Profiling: Heap Dump Analysis IBM JVMs: System Dump of PHD
– -Xdump:system|heap:events=systhrow,filter=java/lang/OutOfMemoryError
– -Xdump:system|heap:events=user
– -Xtrace:maximal=mt,trigger=method{<name>,sysdump|heapdump}
– com.ibm.jvm.Dump.JavaDump()/HeapDump()
– Health Center
Visualization available in IBM Memory Analyzer– Supports both IBM and HotSpot dumps– Provides leak and footprint analysis– Provides heap and application visualization
Oracle JVMs: HPROF dump– -XX:+HeapDumpOnOutOfMemoryError– jmap -dump:format=b– JConsole
© 2014 IBM Corporation19
Java Heap Profiling: Heap Dump Analysis
© 2014 IBM Corporation20
Debugging from Dumps
© 2014 IBM Corporation21
Debugging from Dumps● System dumps and HPROF dumps contain Object fields and values
● System dumps also have thread stacks and local variables● Provides visibility into code flow● Zero ongoing performance overhead● Dump time is ~2s/GB
● Makes it possible to diagnose a much wider set of issues● Exceptions: ability to look at data not exposed in the message● Functional Issues: possible to look at the data being acted on
© 2014 IBM Corporation22
Exceptions● java.net.ConnectException
● Stack trace in Memory Analyzer givesaccess to local variables- host Name = Bailey-W530- port = 100
java.net.ConnectException: Connection refused: connect at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:352) at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:214) at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:201) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:377)
© 2014 IBM Corporation23
Functional Issues● java.lang.OutOfMemoryError allocating a 1,129,665,376 byte object
● The readTokens method references a ByteArrayInputStream with a pos of 2369
● Content of ByteArrayStream around 2369:
● Which converts to 1,129,665,364
at com/ibm/wsspi/security/token/WSOpaqueTokenHelper.readTokens(WSOpaqueTokenHelper.java:924)at com/ibm/wsspi/security/token/WSOpaqueTokenHelper.createTokenHolderListFromOpaqueToken(WSOpaqueTokenHelper.java:844)at com/ibm/ws/sib/security/impl/MessagingEngineIdentityImpl.fromBytesInDomain(MessagingEngineIdentityImpl.java:439)at com/ibm/ws/sib/security/impl/MessagingEngineIdentityImpl.access$200(MessagingEngineIdentityImpl.java:91)at com/ibm/ws/sib/security/impl/MessagingEngineIdentityImpl$2.run(MessagingEngineIdentityImpl.java:299)at com/ibm/ws/sib/security/impl/MessagingEngineIdentityImpl$2.run(MessagingEngineIdentityImpl.java:293)at com/ibm/ws/sib/security/impl/BusUtilities.doInBusDomain(BusUtilities.java:115)
© 2014 IBM Corporation24
Performance
© 2014 IBM Corporation25
Performance Analysis Need to look at each of the layers of the application:
– Platform: Machine and Operating System resources– Runtime: Java heap resources– Application: Inefficient code, synchronization
Recommended to look at Platform and Runtime first– “Easy to resolve”
Biggest gains are in the application and back end responsiveness– OS and Java Runtime are already highly tuned and self adapting
© 2014 IBM Corporation26
OS Level Memory Monitoring
procs -----------memory---------- -swap- --io-- --system-- ---cpu--- r b swpd free buff cache si so bi bo in cs us sy id wa 0 0 17196 679860 1196656 2594884 0 0 1 4 0 0 0 0 100 0 0 0 17196 679868 1196656 2594884 0 0 0 40 1012 43 0 0 100 0 0 0 17196 679992 1196656 2594884 0 0 0 3 1004 43 0 0 100 0
Linux: vmstat {interval} {samples} > vmstat.out
top -H -b -c > top_threads.out
Windows: “perfmon”:Process > Page Faults/secProcess > %Processor TimeNetwork > Output Queue LengthPhysical Disk > Current Disk Queue Length
Can be collected to CSV file
top - 15:46:52 up 178 days, 4:53, 2 users, load average: 0.31, 0.08, 0.02 Tasks: 77 total, 2 running, 74 sleeping, 1 stopped, 0 zombie Cpu(s): 24.6% us, 0.5% sy, 0.0% ni, 74.9% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 5591016k total, 5416896k used, 174120k free, 1196656k buffers Swap: 2104472k total, 17196k used, 2087276k free, 2594884k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8502 user1 25 0 599m 466m 5212 R 99.9 8.5 0:23.92 java 7547 root 15 0 33472 2916 1144 S 0.3 0.1 778:17.98 X 1 root 16 0 656 228 192 S 0.0 0.0 0:07.43 init
© 2014 IBM Corporation27
Garbage Collection Monitoring IBM JVMs:
– -verbose:gc – -Xverbosegclog:<name>,X,Y
Visualization available in GCMV for both IBM and Oracle verbose:gc
Oracle JVMs:– -XX:+PrintGCDateStamps– -XX:+PrintGCDetails– -XX:+PrintGCTimeStamps– -Xloggc:<name>
© 2014 IBM Corporation28
Runtime GC Monitoring IBM Health Center (IBM JVMs*)
– -Xhealthcenter
Used Heap and Pause Times
Allocation Profiling
Oracle Mission Control (HotSpot JVMs)– -XX:+UnlockCommercialFeatures – -XX:+FlightRecorder
Used Heap and Pause Times
Allocation profiling
© 2014 IBM Corporation29
Application method CPU usage IBM Health Center (IBM JVMs*)
– -Xhealthcenter
CPU usage per method
Method call graphs
Oracle Mission Control (HotSpot JVMs)– -XX:+UnlockCommercialFeatures – -XX:+FlightRecorder
CPU usage per method
Method call graphs
© 2014 IBM Corporation30
Application lock usage IBM Health Center (IBM JVMs*)
– -Xhealthcenter
Contention frequency and time
Lock hold time
Oracle Mission Control (HotSpot JVMs)– -XX:+UnlockCommercialFeatures – -XX:+FlightRecorder
Contention frequency and time
Lock hold time
© 2014 IBM Corporation31
Questions?
© 2014 IBM Corporation32
IBM Developer Kits for Javaibm.biz/javasdk
WebShere Liberty Profilewasdev.net
IBM Bluemixibm.com/bluemix
IBM Developer Kits for Node.jsibm.biz/nodesdk
© 2014 IBM Corporation
ibm.biz/javaone2014Visit Booth 5511 to learn about Cloud, DevOps and Mobile solutions
www.ibm.com/developer
Discovernew technical resources.
Developyour coding skills.
Connectwith developers.
© 2014 IBM Corporation34
Copyright and Trademarks
© IBM Corporation 2013. All Rights Reserved.
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corp., and registered in many jurisdictions worldwide.
Other product and service names might be trademarks of IBM or other companies.
A current list of IBM trademarks is available on the Web – see the IBM “Copyright and trademark information” page at URL: www.ibm.com/legal/copytrade.shtml
© 2014 IBM Corporation35