mjprof: monadic approach for jvm profiling

@lifeyx#Devoxx #Performance #AdoptOpenJDK

mjprof : A Monadic Approach for JVM Profiling

Haim YadidPerformize-IT


About Me: Haim Yadid

• 21 Years of SW development experience

• independent Performance Expert

• Consulting R&D Groups

• Training: Java Performance Optimization

• Community leader : Java.IL


Java.IL

• Israeli Java Community and User Group

• Meetup : www.meetup.com/JavaIL/

• Twitter: @java_il

• ~ 400 members

• Meeting every month


Background &

Motivation


The Two Most Important Questions



My wife asks me



What are you doing? When will you be back?

My wife asks me


Profiler

• A Performance analysis tool

• Which answers questions about method calls:

• E.g. : Number of invocation, Average duration,CPU consumptions Contention Memory allocation IO, Cache misses etc…..


Statistical Profiler

• A.k.a. as Sampling Profiler

• Samples the treads activity every several milliseconds

• Not very accurate for short tasks

• Low/Medium overhead

• Do not provide the number of invocations


Instrumenting Profiler

• A.k.a Tracing

• Instruments the code

• Knows when a method starts and when it ends

• High overhead (depends on instrumentation)


Instrumenting Profiler

• A.k.a Tracing

• Instruments the code

• Knows when a method starts and when it ends

• High overhead (depends on instrumentation)

Public void foo(a) {Measure start timeincrease invocation count

bla bla;Measure end time

}

Injected code


jstack - A Poor Man’s Profiler

• The most trivial sampling profiler

• Part of the JDK

• Connects to a running JVM and samples all the threads

• Only once !

• Writes to standard output


And it looks like this

• For each thread we have the following block


And it looks like this

• For each thread we have the following block

"pool-35-thread-1" prio=5 tid=10315b800 nid=0x12c1fe000 waiting on condition [12c1fd000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <107ac0140> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos ..... Locked ownable synchronizers: - None

Thread name Priority Id and native id


Now do this a Million Times!


Now do this a Million Times!

• Tons of threads

• Deep stacks


mjprof


mjprof

• A command line tool

• Acts on jstack output

• Extends it (enriched thread dumps)

• Applies a sequence of operations (monads) to

• Filter

• Aggregate

• Manipulate


Prerequisites

• Java installed

• Set JAVA_HOME (JDK1.7 or later)

• Add ${JAVA_HOME}/bin to path

• Attaching to processes requires JDK and not JRE

• Can run on a headless machine


Installing mjprof

• Git repo : https://github.com/AdoptOpenJDK/mjprof

• Download and unzip unzip mjprof1.0-bin.zip

• Or clone and build mvn install assembly:assembly

• Add mjprof directory to your path


Synopsis and Help (Smoke test)

• Running mjprof without parameters will print help

• It is a good way to see it is functioning


Synopsis and Help (Smoke test)

• Running mjprof without parameters will print help

• It is a good way to see it is functioning

➜ mjprof git:(master) ✗ mjprof Synopsis A list of the following monads concatenated with . … Data sources: jstack/pid,[count],[sleep]/ -Generate dumps using stack path/path/ -Read thread dump from file stdin -Read thread dumps from standard input visualvm/path/ -Read profiling session from xml export of VisualVM

….


Monads

• Pipeline Building blocks

• Monads can

• Collect data

• Filter out certain threads

• Transform threads information

• Aggregate threads information

• Snapshot write state to a file or GUI

• Concatenate with a . (period)


Data Sourcesstdin

file

jstack

jmx


Data Sources

• cat mythreaddump.txt | mjprof…stdin

file

jstack

jmx


Data Sources

• cat mythreaddump.txt | mjprof…

• mjprof path/mythreaddump.txt/

stdin

file

jstack

jmx


Data Sources



• mjprof jstack/19873/

stdin

file

jstack

jmx


Data Sources



• mjprof jstack/19873/

• mjprof jmx/myhost:65327/

stdin

file

jstack

jmx


Monads

contains -contains group

merge

sort

-sort

frame

count

mergedcalles

-frame

top

bottom

trimbelow

namesuf

-fn

-pkg

-fn-prop

jvm -jvm

blocked

locks

-lockswaiting

running withstack

Filters Aggregations Stack frame level

stdout

gui

snapshot

output


Extending mjprof

• mjprof is inherently extensible

• Macros

• A text file named macros.properties

• Contains a sequence of macros

• Plugins

• Annotate your extension with @Plugin

• Compile it and place your jar in the directory plugins under mjprof installation


10 mjprof Recipes


Question: How many threads my process has?



mjprof jstack/1971/path/stack.txt/

jstack/MainClass/



➜ distro mjprof jstack/1971/.count 2014-11-01 10:34:10Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode):

Total number of threads is 1045

mjprof jstack/1971/ . countpath/stack.txt/

jstack/MainClass/


Question: How many jetty thread are working ATM?



mjprof jstack/MyApp/



mjprof jstack/MyApp/ . contains/name,qtp/



➜ 04112014 mjprof jstack/My4tApp/.contains/name,qtp/ 2014-11-04 20:05:03Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode):

"qtp817406040-17-selector-ServerConnectorManager@69b87e8d/3" prio=5 ….3 runnable [0x000000011bc5f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x0000000780209080> (a sun.nio.ch.Util$2) - locked <0x0000000780209070> (a java.util.Collections$UnmodifiableSet)

mjprof jstack/MyApp/ . contains/name,qtp/contains/state,RUN/.



➜ 04112014 mjprof jstack/My4tApp/.contains/name,qtp/ 2014-11-04 20:05:03Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode):

"qtp817406040-17-selector-ServerConnectorManager@69b87e8d/3" prio=5 ….3 runnable [0x000000011bc5f000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method) at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:202) at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:103) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x0000000780209080> (a sun.nio.ch.Util$2) - locked <0x0000000780209070> (a java.util.Collections$UnmodifiableSet)

mjprof jstack/MyApp/ . contains/name,qtp/count.contains/state,RUN/.


Question: Which threads are running my logic ?


Question: Which threads are running my logic ?

2014-11-04 20:11:55Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.0-b70 mixed mode):

"Indexing daemon" prio=5 tid=0x00007fb306bf4800 nid=0x6307 waiting on condition [0x000000011c171000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at com.myapp.IndexManagement$1.run(IndexManagement.java:148)

"main" prio=5 tid=0x00007fb303809000 nid=0x1303 in Object.wait() [0x0000000105500000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:502) at com.myapp.MyAppApp.main(MyAppApp.java:18)

mjprof jstack/MyApp/ . contains/stack,myapp/


Question: Stack-trace is cluttered, how do I shorten it ?



mjprof jmx/MyApp/ .



mjprof jmx/MyApp/ . frame/mycompany/



mjprof jmx/MyApp/ . frame/mycompany/-frame/grizzly/



mjprof jmx/MyApp/ . frame/mycompany/-frame/grizzly/top/3/


Question: How can I group threads by state ?


mjprof jmx/MyApp/ . group/state/ gui.

Question: How can I group threads by state ?


Question: How do I aggregate several thread dumps



mjprof jmx/MyApp,10,1000/



mjprof jmx/MyApp,10,1000/ .group/state/

merge



mjprof jmx/MyApp,10,1000/ .group/state/

merge .


Question: How can I combine thread dumps from several processes?


Question: How can I combine thread dumps from several processes?

mjprof jmx/myhost1.com:5467/ .

group/state/ gui.jmx/myhost2.com:5467/ .

http://myhost1.com

http://myhost2.com


How do I measure thread pool CPU consumption?



mjprof jmxc/MyApp/



mjprof jmxc/MyApp/ . contains/name,COFFEE/


"*" count=7 tid=* cpu_ns=1164536000 wall_ms=4804 %cpu=24.24 java.lang.Thread.State: TIMED_WAITING100.00% [7/7]\ at Thread.run(Thread.java:744)100.00% [7/7] \ at Sleeper.run(Sleeper.java:51)100.00% [7/7] \ at Sleeper.loopForever(Sleeper.java:37)100.00% [7/7] \ at Sleeper.longerStackTrace(Sleeper.java:32)100.00% [7/7] \ at Sleeper.longerStackTrace(Sleeper.java:32)100.00% [7/7] \ at Sleeper.longerStackTrace(Sleeper.java:32)100.00% [7/7] \ at Sleeper.longerStackTrace(Sleeper.java:32)100.00% [7/7] \ at Sleeper.longerStackTrace(Sleeper.java:32)100.00% [7/7] \ at Sleeper.longerStackTrace(Sleeper.java:30)100.00% [7/7] \ at Recurse.singleCycle(Recurse.java:16)100.00% [7/7] \ at Recurse.doRecursion(Recurse.java:25)100.00% [7/7] \ at Recurse.doRecursion(Recurse.java:25)


mjprof jmxc/MyApp/ . contains/name,COFFEE/-pkg group..


Question: Which threads are blocked by the lock I hold


Question: Which threads are blocked by the lock I hold

mjprof path/threaddump.txt/ .blockedcontains/stack,0x00000007aab51b38/

.


Question: What is the overall locking state of my application


Question: What is the overall locking state of my application

mjprof path/threaddump.txt/ .contains/stack,myapp/locks

.

"CPU3_5" prio=5 tid=0x00007fc4f4889800 nid=0x5703 waiting for monitor… java.lang.Thread.State: BLOCKED (on object monitor) - waiting to lock <0x00000007aad98988> (a java.lang.Object)

"CPU3_4" prio=5 tid=0x00007fc4f404a800 nid=0x5503 waiting for monitor… java.lang.Thread.State: BLOCKED (on object monitor) - waiting to lock <0x00000007aad98988> (a java.lang.Object)

"CPU3_3" prio=5 tid=0x00007fc4f4049800 nid=0x5303 waiting on … java.lang.Thread.State: TIMED_WAITING (sleeping) - locked <0x00000007aad98988> (a java.lang.Object)


Thanks / Contact Me

http://il.linkedin.com/in/haimyadid

[email protected]

www.performize-it.com

blog.performize-it.com

https://github.com/lifey

@lifeyx





https://github.com/lifey


mjprof: monadic approach for jvm profiling

Software