debugging your production jvm
DESCRIPTION
This is a talk I did for JavaOne 2009. The focus of the talk was memory management and system monitoring with freely available tools that are in the jdk or open source.TRANSCRIPT
![Page 2: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/2.jpg)
Abstract
> Learn the tools and techniques used to monitor, trace and debugging running Java applications.
2
![Page 3: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/3.jpg)
Agenda
> Java Memory Management> Memory Management Tools
• Command-line Tools• VisualVM
> Btrace> Summary> Resources
3
![Page 4: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/4.jpg)
Agenda
> Java Memory Management> Memory Management Tools
• Command-line Tools• VisualVM
> Btrace> Summary> Resources
4
![Page 5: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/5.jpg)
Donʼt Worry…
> C (malloc / free)> C++ (new / delete)> Java (new / gc)> Memory Allocation / Deallocation in Java is
automatic
5
![Page 6: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/6.jpg)
Object Lifetimes
> Most objects live very short lives• 80-98% of all newly allocated objects die
• within a few million instructions• Before another megabyte is allocated
> Old objects tend to live a long…. time
![Page 7: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/7.jpg)
Memory Spaces
> New Space• Where all objects are new (sort of…)
> Old Space• Where all objects are old
New Space Old Space
![Page 8: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/8.jpg)
Memory Spaces
> New Space Division• Eden
• Where all objects are created
• Survivor Space 0• Provide object aging
• Survivor Space 1
Eden Old SpaceSS0 SS1
![Page 9: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/9.jpg)
Perm Spaces
> Permanent Space• class information• static information
Eden Old SpaceSS0 SS1 Perm Space
![Page 10: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/10.jpg)
GC Responsibility
> Heap Walking from GC Roots> Mark / Sweep
• Garbage detection (mark)• Sort out the live ones from the dead ones• Reference counting
• Garbage reclamation (sweep)• Make space available
![Page 11: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/11.jpg)
Minor Garbage Collection
> Minor gc (scavenge)• When eden is “full” a minor gc is invoked• Sweeps through eden and the current survivor
space, removing the dead and moving the living to survivor space or old
• Ss0 and ss1 switch which is “current”• A new tenuring age is calculated
![Page 12: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/12.jpg)
Major Garbage Collection
> Major gc• When old is “full”• All spaces are garbage collected including perm
space• All other activities in the jvm are suspended
![Page 13: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/13.jpg)
![Page 14: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/14.jpg)
Agenda
> Java Memory Management> Memory Management Tools
• Command-line Tools• VisualVM
> Btrace> Summary> Resources
14
![Page 15: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/15.jpg)
Java Memory Tools
> JPS• Getting the Process ID (PID)
> Jstat• jstat -gcutil <pid> 250 7
![Page 16: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/16.jpg)
Looking at the Heap
> %JAVA_HOME%/bin/jmap – histo:live <pid>• Looking at all the “live” objects
> %JAVA_HOME%/bin/jmap – histo <pid>• Looking at all objects
> The difference between is the list of unreachable objects
![Page 17: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/17.jpg)
Taking a Heap Dump> %JAVA_HOME%/bin/jmap –
dump:live,file=heap.out,format=b <pid>• Dumps the Heap to a file
> JConsole
![Page 18: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/18.jpg)
JHat
> %JAVA_HOME%/bin/jhat <filename>• Starts a web server to investigate the heap
> Queries• Show instance count for all classes• Show Heap Histogram• Show Finalizer• Use the Execute Object Query Language (OQL)
• select s from java.lang.String s where s.count >=100
![Page 19: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/19.jpg)
JMX – Looking at Flags
![Page 20: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/20.jpg)
MAT – Memory Analyzer Tool
![Page 21: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/21.jpg)
VisualVM
> Open Source All-in-One Java Troubleshooting tool> https://visualvm.dev.java.net/
21
![Page 22: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/22.jpg)
Visualgc> visualgc <pid>> Visual Garbage Collection
Monitoring• a graphical tool for monitoring the
HotSpot Garbage Collector, Compiler, and class loader. It can monitor both local and remote JVMs.
![Page 23: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/23.jpg)
23
DEMO
![Page 24: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/24.jpg)
Agenda
> Java Memory Management> Memory Management Tools
• Command-line Tools• VisualVM
> Btrace> Summary> Resources
24
![Page 25: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/25.jpg)
BTrace
> dynamically bytecode instrumenting (BCI) • read / not write• probe-based• observe running Java applications
> Integration with DTrace on Solaris> http://btrace.dev.java.net> visualvm plugin
25
![Page 26: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/26.jpg)
BTrace Tools
26
Target JVM
BTrace Agent
VisualVM +BTrace Plugin
BTrace command-line
![Page 27: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/27.jpg)
BTrace Terminology
> Probe Point• “location” or “event” at which tracing statements are
executed> Trace Actions
• statements which are executed whenever a probe fires
> Action Methods• static methods which define a trace
27
![Page 28: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/28.jpg)
Probes and Actions
> Probe Targets• method entry / exit• line number• exceptions
• return from method• exception throw (before)
• synchronization entry / exit> Actions
• static methods in trace class28
![Page 29: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/29.jpg)
BTrace Restrictions
> no new objects> no new arrays> no exceptions> no outer, inner, nested or
local classes> no synchronization blocks> no loops> no interfaces
29
![Page 30: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/30.jpg)
Tracing
> Annotations• com.sun.btrace.annotations• @BTrace
• denotes a btrace class
> Probe Points• @OnMethod• @OnTimer• @OnEvent• @OnExit
30
![Page 31: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/31.jpg)
Simple Example: Looking for Object Size
31
![Page 32: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/32.jpg)
Simple Example: Looking for Object Size
32
![Page 33: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/33.jpg)
BTrace Run Options
> BTrace Command-Line• btrace
• runs btrace tool (and compiles btrace script if necessary)
• btracec• btrace script compiler
• btracer• convenience script to start java project with tracing enabled at
application startup
> VisualVM + BTrace Plugin
33
![Page 34: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/34.jpg)
Testing the Probe
> Start Target Application• java -jar java2demo.jar
> Get the PID• jps
> Inject Probe• btrace <pid> Sizeof.java
34
![Page 35: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/35.jpg)
BTrace + JMX Script
35
![Page 36: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/36.jpg)
BTrace + jstat
36
![Page 37: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/37.jpg)
ThreadLocal
37
![Page 38: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/38.jpg)
BTrace Events
38
![Page 39: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/39.jpg)
Checking Synchronization Entry / Exits
39
![Page 40: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/40.jpg)
Tracing Opportunities
> Thread Monitors> Socket / Web Services> Object Creation and Size> File Access
40
![Page 41: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/41.jpg)
41
DEMO
![Page 42: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/42.jpg)
Known Solutions with BTrace
> Terracotta • concurrency issue
> Hibernate / Atlassian Issue• thread / session management issue• http://jira.atlassian.com/browse/CONF-12201
42
![Page 43: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/43.jpg)
Summary
> jmap, jhat, jconsole, jstat• tools already in the JDK bin directory
> VisualVM• swiss army knife for JVM process monitoring and
tracing> BTrace
• dynamic tracing tool
43
![Page 44: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/44.jpg)
Resources
> Performance and Troubleshooting• http://java.sun.com/performance/reference/
whitepapers/6_performance.html• http://java.sun.com/javase/6/webnotes/trouble/TSG-
VM/html/docinfo.html> VisualVM
• https://visualvm.dev.java.net/> BTrace
• https://btrace.dev.java.net/
44
![Page 45: Debugging Your Production JVM](https://reader033.vdocument.in/reader033/viewer/2022051514/54b73deb4a7959be4c8b4842/html5/thumbnails/45.jpg)
Ken Sipehttp://kensipe.blogspot.com
twitter: @kensipe