java flight recorder behind the scenes
DESCRIPTION
Presentation from JavaOne 2013TRANSCRIPT
![Page 1: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/1.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 1
![Page 2: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/2.jpg)
Java Flight Recorder Behind the Scenes Staffan Larsen Java SE Serviceability Architect [email protected] @stalar
![Page 3: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/3.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 3
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.
![Page 4: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/4.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 4
Java Flight Recorder § Tracer and Profiler § Non-intrusive § Built into the JVM itself § On-demand profiling § After-the-fact capture and analysis § First released in 7u40
Photograph by Jeffrey Milstein http://butdoesitfloat.com/In-the-presence-of-massive-gravitational-fields-this-perfect
![Page 5: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/5.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 5
Tracer and Profiler § Captures both JVM and application data
– Garbage Collections – Synchronization – Compiler – CPU Usage – Exceptions – I/O
§ Sampling-based profiler – Very low overhead – Accurate data
![Page 6: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/6.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 6
Non-Intrusive § Typical overhead in benchmarks: 2-3% (!) § Often not noticeable in typical production environments § Turn on and off in runtime § Information already available in the JVM
– Zero extra cost
Photograph: Andrew Rivett http://www.flickr.com/photos/veggiefrog/3435380297/
![Page 7: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/7.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 7
Built into the JVM itself § Core of JFR is inside the JVM § Can easily interact with other JVM subsystems § Optimized C++ code § Supporting functionality written in Java
![Page 8: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/8.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 8
On-Demand Profiling § Start from Java Mission Control
– Or from the command line § Easily configure the amount of information to
capture § For a profile, a higher overhead can be
acceptable § When done, no overhead § Powerful GUI for analysis
![Page 9: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/9.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 9
After-the-Fact Analysis § In its default mode, very low overhead § Designed to be always-on § Uses circular buffers to store data
– In-memory or on-disk § When an SLA breach is detected, dump the current buffers § Dump will have information leading up to the problem
n
n+1
n+2 n+3
n+4
![Page 10: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/10.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 10
![Page 11: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/11.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 11
Agenda
§ Overview of JFR
§ Demo!
§ Configuration topics
§ Implementation details
![Page 12: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/12.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 12
Configuration § Enable
-‐XX:+UnlockCommercialFeatures -‐XX:+FlightRecorder
§ Start -‐XX:StartFlightRecording=filename=<path>,duration=<time>
§ Or jcmd <pid> JFR.start filename=<path> duration=<time>
![Page 13: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/13.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 13
Advanced Configuration Per Recording Session Max age of data maxage=<time> Max size to keep maxsize=<size>
Global Settings (-XX:FlightRecorderOptions) Max stack trace depth stackdepth=<n> (default 64) Save recording on exit dumponexit=true Logging loglevel=[ERROR|WARN|INFO|
DEBUG|TRACE] Repository path repository=<path>
![Page 14: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/14.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 14
Recording Sessions § Recordings can specify exactly which information to capture
– ~80 events with 3 settings each § But: two preconfigured settings
– “default”: provides as much information as possible while keeping overhead to a minimum
– “profile”: has more information, but also higher overhead
§ You can configure your own favorites in Mission Control
![Page 15: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/15.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 15
Many Simultaneous Recording Sessions § This works great § Each session can have its own settings § Caveat: If there are multiple sessions all of them get the union of the
enabled events – Ex: If event A is enabled in on recording, all recordings will see event A – Ex: If event B has two different thresholds, the lower value will apply
![Page 16: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/16.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 16
JVM
How Is It Built?
§ Information gathering – Instrumentation calls all over the
JVM – Application information via Java
API
§ Collected in Thread Local buffers ⇢ Global Buffers ⇢Disk
§ Binary, proprietary file format § Managed via JMX
JFR
Thread Buffers
Disk
JMX Java API
Global Buffers
GC Runtime Compiler
![Page 17: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/17.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 17
“Everything Is an Event” § Header § Payload
– Event specific data
Event ID
End Time
Start Time
Thread ID
StackTrace ID
Payload
Header
Optional
Size
![Page 18: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/18.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 18
Event Types § Instant
– Single point in time – Ex: Thread starts
§ Duration – Timing for something – Ex: GC
§ Requestable – Happens with a specified frequency – Ex: CPU Usage every second
![Page 19: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/19.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 19
Event Meta Data § For every event
– Name, Path, Description
§ For every payload item – Name, Type, Description, Content Type
![Page 20: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/20.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 20
“Content Type” § Describes the semantics of a value § Used to correctly display the value in the UI
Content Type Displayed as Bytes 4 MB Percentage 34 % Address 0x23CDA540 Millis 17 ms Nanos 4711 ns
![Page 21: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/21.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 21
Event Definition in Hotspot <event id="ThreadSleep"
path="java/thread_sleep" label="Java Thread Sleep" ...>
<value field="time" type="MILLIS" label="Sleep Time"/>
</event>
§ XML definitions are processed into C++ classes
![Page 22: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/22.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 22
Event Emission in Hotspot JVM_Sleep(int millis) { EventThreadSleep event; ... // actual sleep happens here event.set_time(millis); event.commit(); }
§ Done! Data is now available in JFR
![Page 23: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/23.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 23
Thread Park <event id="ThreadPark" path="java/thread_park" label="Java Thread Park" has_thread="true" has_stacktrace="true" is_instant="false"> <value type="CLASS" field="klass" label="Class Parked On"/> <value type="MILLIS" field="timeout" label="Park Timeout"/>
<value type="ADDRESS" field="address" label="Address of Object Parked"/> </event>
![Page 24: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/24.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 24
Thread Park UnsafePark(jboolean isAbsolute, jlong time) { EventThreadPark event; JavaThreadParkedState jtps(thread, time != 0); thread-‐>parker()-‐>park(isAbsolute != 0, time);
if (event.should_commit()) { oop obj = thread-‐>current_park_blocker(); event.set_klass(obj ? obj-‐>klass() : NULL); event.set_timeout(time); event.set_address(obj ? (TYPE_ADDRESS)obj : 0); event.commit(); } }
![Page 25: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/25.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 25
Event Graph
![Page 26: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/26.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 26
Event Details
![Page 27: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/27.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 27
Buffers § “Circular” § Designed for low contention
Thread Local
Buffers
Global Buffers
Disk Chunk
Repository
JVM
Java
![Page 28: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/28.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 28
Filtering Early § Enable/disable event § Thresholds
– Only if duration is longer than X § Enable/disable stack trace § Frequency
– Sample every X
![Page 29: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/29.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 29
File Format § Self-contained
– Everything needed to parse an event is included in the file – New events instantly viewable in the UI
§ Binary, proprietary § Designed for fast writing § Single file, no dependencies
Event Records Event Definitions … Header
![Page 30: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/30.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 30
Dynamic Runtime and Long-Running Recordings § Can’t leak memory
– Can’t aggregate information eternally – Can’t keep references that prohibits class unloading
§ Dynamic Runtime – Classes can come and go – Threads can come and go
§ Solutions: Constant Pools, Checkpoints
![Page 31: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/31.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 31
Problem: Many Events Reference Classes § If every event contained the class name as a string, we would waste
lots of space § Solution: Class IDs
class Klass { … u8 _trace_id; … }
17 event id
class count
java.lang.Integer 2
… 17 event id
class count
4711 2
…
![Page 32: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/32.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 32
Problem: When Do We Write the Class IDs? § IDs need to be part of the file § Classes can be unloaded at any time
– Class may not be around until end of recording § Solution: write Class ID when classes are unloaded
![Page 33: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/33.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 33
Problem: Size of the Class List § Many classes are loaded, not all are referenced in events, we want to
save space § Solution: when a class ID is referenced, the class is also “tagged”
– Write only tagged classes in the JFR file
#define CLASS_USED 1 void use_class_id(Klass* const klass) { klass-‐>_trace_id |= CLASS_USED; }
![Page 34: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/34.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 34
Problem: Leaking Memory § Over time many classes will be tagged, the size of the class list will
increase § Solution: reset the tags each time a class list is written to disk § We call this a “Checkpoint”
§ A recording file may contain many class lists, each one is only valid for the data immediately preceding it
![Page 35: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/35.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 35
Constant Pools § The Class List is a special case of a Constant Pool
§ Classes § Methods § Threads § Thread Groups § Stack Traces § Strings
class_pool.lookup(4711) ➞ java.lang.Integer method_pool.lookup(1729) ➞ java.lang.Math:pow()
![Page 36: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/36.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 36
Checkpoints § At regular intervals, a “checkpoint” is
created in the recording § Has everything needed to parse the
recording since the last checkpoint
checkpoint = events + constant pools + event meta-data
events
constant pools
meta-data
checkpoint
JFR File
![Page 37: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/37.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 37
Optimizations § Fast Timestamps
– Fast, high resolution CPU time where available – Invariant TSC instructions
§ Stack Traces – Each event stores the thread’s stack trace – Pool of stack traces
![Page 38: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/38.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 38
Differences vs. JRockit § I/O: File path, Socket address § Exceptions § Reverse call trace view in Mission Control § Easier configuration in Mission Control § Deeper (configurable) stack traces § Internal JVM differences: GC
![Page 39: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/39.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 39
More Information § Whitepaper
http://www.oracle.com/missioncontrol § User Guide
http://docs.oracle.com/javase/7/docs/technotes/guides/jfr/index.html
§ Forum https://forums.oracle.com/community/developer/english/java/java_hotspot_virtual_machine/java_mission_control
![Page 40: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/40.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 40
-‐XX:+UnlockCommercialFeatures -‐XX:+FlightRecorder
Remember
![Page 42: Java Flight Recorder Behind the Scenes](https://reader034.vdocument.in/reader034/viewer/2022051411/54623f6aaf79597c138b4953/html5/thumbnails/42.jpg)
Copyright © 2013, Oracle and/or its affiliates. All rights reserved. 42