diagnose apps back to basics
TRANSCRIPT
![Page 1: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/1.jpg)
Diagnose Java apps on JVM - Back to Basics
Imran Bohoran@imranbohoran
![Page 2: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/2.jpg)
Formalities
● Who am I○ Developer at Unruly
● What do I like○ Coding (obviously)○ Diagnosing application failures○ Discovering new tools and tricks I never knew of
![Page 3: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/3.jpg)
What basics?
● Various tools at our disposal● Do the tools tell us what we need to find out?● Tools are not a bad thing - they are great.● But going back to some basics can help
more○ What do we have available for us from the JVM
![Page 4: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/4.jpg)
The application is down● Who has experienced it?● Is it fun?
![Page 5: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/5.jpg)
The application is down
● The load is high● The JVM is not responsive● Everyone’s breathing down your neck
![Page 6: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/6.jpg)
What do you do?
● Do you panic?● Do you run around telling everyone?● Do you try to work out what’s going on using
your favourite tool?● Do you restart the app?● Do you inspect the logs?● Do you take a thread dump(s)?
![Page 7: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/7.jpg)
Thread dumps
● Who has taken a thread dump or stared at them?
● What are they?● How do you take thread dumps
○ jstack <pid>○ kill -SIGQUIT or kill -3
![Page 8: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/8.jpg)
Thread dumps
● Application Threads● VM Threads
![Page 9: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/9.jpg)
A Thread dump - Application threads"http-nio-8080-ClientPoller-1" #23 daemon prio=5 os_prio=31 tid=0x00007fbf1c7e0000 nid=0x6b03 runnable [0x000000012aa2f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000782cb26f0> (a sun.nio.ch.Util$2)
- locked <0x0000000782cb26e0> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000782938878> (a sun.nio.ch.KQueueSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.tomcat.util.net.NioEndpoint$Poller.run(NioEndpoint.java:1179)
at java.lang.Thread.run(Thread.java:745)
![Page 10: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/10.jpg)
A Thread dump - VM threads"C2 CompilerThread0" #5 daemon prio=9 os_prio=31 tid=0x00007fbf19025800 nid=0x4903 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007fbf1c011800 nid=0x471f waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007fbf19024800 nid=0x3503 in Object.wait() [0x0000000123cec000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
"Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007fbf19024000 nid=0x3303 in Object.wait() [0x0000000123be9000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
"VM Thread" os_prio=31 tid=0x00007fbf1901f000 nid=0x3103 runnable
"GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007fbf1901d800 nid=0x2103 runnable
![Page 11: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/11.jpg)
Thread dump - Important elements"C2 CompilerThread0" #5 daemon prio=9 os_prio=31 tid=0x00007fbf19025800 nid=0x4903 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
● Thread state ○ RUNNABLE○ WAITING○ TIMED_WAITING○ BLOCKED
● Thread ID - tid● Native ID - nid● Thread name
![Page 12: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/12.jpg)
How does this help?
● They can tell us all threads that are running, blocking and waiting
● They can point to code lines that are blocking threads
● They can tell us if we are running out of memory○ VM threads busy○ With heap utilisation summary (kill -3)
![Page 13: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/13.jpg)
How does this help?
● Find out what thread is eating up the most CPU○ Individual jvm thread is mapped to its own process○ top -H -p <pid>○ Other ways
■ topthreads plugin for jconsole■ JMC JMX view
![Page 14: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/14.jpg)
Whats hard about thread dumps
● Reading them and understanding them● The can be loooong● What can we do
○ spend time in understanding on how to interpret them
○ use tools
![Page 15: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/15.jpg)
Tools that are helpful
● IBM Thread and Monitor Dump Analyzer for Java
(only if you want to get a thread dump from Visual VM - not sure why though)
![Page 16: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/16.jpg)
Closing thoughts on Thread dumps
● Application logs○ Log thread id/name on log statements○ Separate them from stdout
● Look at them more to understand your app and dig deeper on problems
● Automate them
![Page 17: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/17.jpg)
Show of hands..
Whats the most common problem you have hit on a JVM app
![Page 18: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/18.jpg)
OOME
● Not enough heap size● PermGen/Metaspace size● GC Overhead limit
![Page 19: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/19.jpg)
Lets talk about GC
● Best source of information for GC○ GC Logs
● How do you get them○ -verbose:gc, -XX:+PrintGCDateStamps, -XX:
+PrintGCTimeStamps, -Xloggc:● Can I set them dynamically
○ jinfo -flags <flag>
![Page 20: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/20.jpg)
What can a GC log tell you
● How long it takes to young and old GC● Workout allocation rates● Premature promotions● Memory leaks
![Page 21: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/21.jpg)
Its all text - show me tools● GC Viewer - https://github.com/chewiebug/GCViewer
● HP Jmeter - https://h20392.www2.hp.com/portal/swdepot/displayProductInfo.do?productNumber=HPJMETER
● IBM Monitoring and Diagnostics Tools - http://www.ibm.com/developerworks/java/jdk/tools/gcmv
● JClarity - http://www.jclarity.com/censum/
![Page 22: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/22.jpg)
What’s the next best thing
● Heap dumps● Automate them
○ -XX:+HeapDumpOnOutOfMemoryError○ -XX:+HeapDumpPath
● To manually get heap dumps○ jmap
![Page 23: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/23.jpg)
I have a heap dump - now what
local profiling (development only - production needs $$$£££)
![Page 24: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/24.jpg)
When looking at GC issues
● Is your heap sizing sensible● Your code is broken● Frameworks are not perfect● You have no reason to use finalizers
![Page 25: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/25.jpg)
To wrap things up..
● Know what your JVM has to offer● Third party tools are great, but fundamental
JVM tools can be more efficient and beneficial (and cheaper)
● Know your tools (get to know them, practice)● Add metrics to your application● Monitor the JVM
○ Standard and custom JMX
![Page 26: Diagnose apps back to basics](https://reader034.vdocument.in/reader034/viewer/2022042701/55a69e1d1a28abda7d8b4856/html5/thumbnails/26.jpg)
Question?