experimental tools for serviceability
TRANSCRIPT
S ess ion 3280
Experimental Tools for Serviceability
Tony Printezis Kenneth RussellL ecturer Member of T ech S taffUnivers ity of G lasgow S un Microsystems
S ess ion 3280 2
Overall Presentation Goal
G et an ins ight into examining, debugging, and profiling different aspects of a J ava™ technology-based V irtual Machine (J V M™ /J ava V M) us ing two new experimental tools
S ess ion 3280 3
Learning Objectives
• A s a result of this presentation, you will be convinced that:– J ava V M debugging is hard!– C urrent debugging tools are not always
adequate– T here is a need for specialised tools to help
the J ava V M implementor...– ...which might also be used to help application
programmers too
S ess ion 3280 4
Speakers’ Qualifications
• T ony P rintezis– L ecturer, Univers ity of G lasgow– He has been writing garbage collectors
for J ava V Ms for four years
• K enneth R ussell– Member of tech staff, S un Microsystems– He has worked on the J ava HotS pot™
runtime and compiler for over two years
S ess ion 3280 5
Bug Hunting Ain’t Fun!
How many all-nighters have you pulled because of a bug in your native code?
Now imagine debugging the G C or the J ava V M itself...
S ess ion 3280 6
“Java VM Debugging Facility”
S ess ion 3280 7
Some Proper Debugging Tools!
• G C spy– T ony P rintezis – Heap visualisation for J ava V Ms• L ow–impact, high–level visualisation
• J ava HotS pot™ S erviceability A gent– K enneth R ussell– E xaminer and debugger for J ava HotS pot V M• P ost–mortem analys is of J ava V M
S ess ion 3280 8
GCspy
• A daptable heap visualisation framework– Incorporated into several J ava V Ms
• D eveloped to visualise the behaviour of G arbage C ollectors (G C s )
• S calable– L arge heap s izes– V ery long–running applications
• D ynamic– Works while J ava V M is executing
S ess ion 3280 9
Why Heap Visualisation?
• G C s have (almost) chaotic behaviour!– O ne small change in the application behaviour
can have a large effect in the G C performance
• G C debugging and profiling facilities– S till very primitive
• V isualisation provides a convenient way to get an ins ight into the G C ’s behaviour– D ebugging and profiling
S ess ion 3280 10
Visualisation Granularity
• C oarse granularity– Heap blocks , not objects– S maller data volumes– F aster to collect / send / store data
• B ut...– L ower level of detail– Information loss
• C an adjust block s ize to deal with scalability and screen s ize limitations
S ess ion 3280
S ess ion 3280 12
Client–Server Model
• C lient–S erver– J ava V M is the server– V isualiser is the client
• F lexibility– C an be run as different processes– C an be run on different machines– C an connect / disconnect the visualiser
during J ava V M execution
S ess ion 3280 13
Portability
• F ramework is J ava V M and G C independent– But (small) modifications to the J ava V M are
needed
• V isualiser operates over some abstractions
• Need to map the G C ’s s tate to those abstractions– A G C spy D river does this
• O nly the server–s ide of the framework needs to be customised
S ess ion 3280 14
Architecture
Visualiser
SpaceRenderer
0
SpaceRenderer
1
Space 0 Space 1
Client Infrastructure
Stream 0
Stream 1
Stream 0
Stream 1
Stream 2
Comms
Java Application
Java VM
Comms
Server Infrastructure
Used
Roots
Objects
Cards
Marked
S-SDriver
M&CDriver
S-S GC M&C GC
Data Collection
Socket
ClientServer
S ess ion 3280 15
Performance Impact
• D ata collection can take a long time– S everal seconds for each G C in the case
of large heaps
• J ava V M performance is affected
• No performance penalty when the visualiser is not connected!
S ess ion 3280 16
Trace Storing / Replaying
• G oal– S tore transmiss ions from a J ava V M– R eplay them at a later time
• R eplay tool behaves exactly like the J ava V M
• C ompact trace files– C ompress ion
S ess ion 3280
S ess ion 3280 18
Current Implementations
• T hree J ava™ technology–based V Ms– J ava HotS pot™ V M– S un’s R esearchV M– IBM’s J ikes R V M
• A t least eight different G C s
• O ther systems too– Not only limited to J ava V Ms!
S ess ion 3280 19
Future Work
• B etter, more flexible visualisations
• O ther visualisation styles– e.g. S catter plots , Histograms
• R ewind facilities
• C onditional breakpoints
S ess ion 3280 20
Java HotSpot Serviceability Agent
• E xaminer and debugger for J ava HotS pot virtual machine
• O riginally des igned for post–mortem analys is
• R uns no code in target J V M
• R ecovers high–level J ava programming language s tate
S ess ion 3280 21
Motivation
• J ava HotS pot V M group needed better debugging tools
• C urrent debuggers not targeted toward debugging the J V M– C /C ++ debuggers• R equire debug info not present in product• D o not understand J ava language, objects
– J ava programming language debuggers• G ather information via J V M D I• D o not work with crashed J ava V Ms
S ess ion 3280 22
Mechanisms
• J ava HotS pot S erviceability A gent (S A )– Understands layouts of key HotS pot data
s tructures• T hread lis t• S tack frames
– C ontains code to walk these data s tructures
• D ebugger tuned for J ava HotS pot V M
• R ecovers same information as J V M D I / J D WP - based debuggers
S ess ion 3280 23
Features
• J ava programming language A P Is– Implemented almost entirely in J ava
programming language• A ll the way down to “read process memory”• E nables cross–O S and cross–C P U debugging
– C an be bound to existing J D I interfaces in J P D A (in progress ; more later)
S ess ion 3280 24
Features
• F ine–grained heap analys is– O bject–by–object iteration– Iteration through objects
• S tack walking– A ccess to local variables , express ion
stack, monitors– F allback mechanisms for analyzing crashes
in interpreter or in compiled code
S ess ion 3280 25
Architectural Diagram
TargetHotSpot
JVM
dbx
import module
SA'sJava VM
Dbx APIs Socket
/proc
• Note that no code is being run in target J ava V M– C urrent J P D A runs J V M D I-based code there
S ess ion 3280 26
Evolution
• F irs t debugger– A ttach to running J V M or open core file– E xamine all J V M state at very low level– P rocess suspended during examination
• T ools for developers of J V M™ technology– S tack memory dump with annotations– O bject inspection
• E nd user tools– Heap dumps, liveness analys is
S ess ion 3280
S ess ion 3280 28
Evolution
• F irs t debugger– Most useful to developers of J ava HotS pot
technology– Most useful on the
S olaris™ O perating E nvironment• Built on dbx• “E scape hatch” to see non–J V M state
– P orted to Win32– No interaction with process• No suspend, resume• No breakpoint setting, exception catching
S ess ion 3280 29
Observation
• S A built on very low–level debugging primitives– “read process memory”– “fetch thread register set”
• S ame primitives as C /C ++ debuggers
• Why not extend to look at more than just J ava programming language s tate?
S ess ion 3280 30
Extensions
• S A A P Is extended:– F etch all O S –level threads– F etch all dynamically loaded objects
• More platform–dependent “J ava code”:– C - level s tack walking (all frames on stack)– P ars ing of C /C ++ debug information
• E nable complete process examination system to be written in J ava programming language
S ess ion 3280 31
Extensions
• F urther extens ions:– R esume/suspend process– P oll for event (seg fault, etc.)– S et breakpoint at machine instruction
• R elatively little additional code
• E nable mostly–complete C /C ++ debugger to be written in J ava programming language
S ess ion 3280 32
Extensions
• C ommunication with J V M D I module– S tart with -X run– E nables J ava programming language breakpoint
setting and exception catching
• T his is the only code run in target J ava V M
• Most functionality available without it
S ess ion 3280 33
Observation
• Now have all primitives necessary to build both C /C ++ and J ava programming language debuggers
• S ystem was originally des igned for post–mortem analys is of J ava V M
• Ideally posed for building mixed–language debugger– C an handle crashes in native code gracefully
S ess ion 3280 34
Current Status
• E xperimental C /C ++ and J ava programming language debugger has been built– Written in J ava programming language
• S upports breakpoints in both C and “J ava code” (J ava™ technology–based code)
• S upports exception catching
• D egrades gracefully– Most of debugger s till works even if target
process crashes
S ess ion 3280
S ess ion 3280 36
Availability
• S A is currently available to source licensees of J ava HotS pot™ virtual machine
S ess ion 3280 37
Future Work
• C omplete ports to all S un–supported platforms
• J D I binding in development– A llow current tools to access J ava programming
language state of J ava HotS pot core files via S A
• D iscuss prototype C /C ++ debugging A P Is with tool vendors– O ptional extens ions to J D I– E nable all tool vendors to build
mixed–language debuggers
S ess ion 3280 38
Presentation Summary
• J ava V M debugging is hard!
• We presented two new debugging tools– Heap visualisation with G C spy– P ost–mortem analys is with the J ava HotS pot
S erviceability A gent
• T hey are used by J ava V M implementors
• T hey operate at two different levels
• T hey improve debugging effectiveness
S ess ion 3280 39
Call to Action
• Need feedback on these tools and A P Is– Whether they would be useful– F eatures which are needed– S hould we ship them (unsupported)
• F eedback on G C spy– [email protected]
• F eedback on the J ava HotS pot S erviceability A gent– sa- [email protected]
S ess ion 3280
S ess ion 3280