open source debugging v1.3.2

by Matthew McCullough of Ambient Ideas, LLC

Debugging ToolsOpen Source

my favorite apps to ease

debugging pain

Matthew

Quick bits about me...

-Denver-Open source architect

-Author of Maven Refcard

-NFJS speaker

-Mentor

OSS Debug Tools

OSS Debug Tools

Web Service

JavaScriptFilesystem

Java

Network

Not

Instead

Why open source

debugging tools?

Today’s Deal

- 90 minute investment

- Hours back in return

NetworkCentric

OSS Debug Tools

Web Service

JavaScriptFilesystem

Java

Network

We’re having a problem on the web

site...web service...whatever

you call it.

It returns the correct <pointy>

code stuff for some folks but not

others.

Purpose‣ Command line URL tasks.‣ Test web services.‣ Inspect raw HTML pages.

remove browser magic

www.microsoft.com

more advanced usage

-X [action]Selects an HTTP verb (POST,GET, PUT, DELETE).

cURL flags

conditional content presentation

-H [HTTP header]Set the Accept or Content-Type headers(e.g. text/xml, or application/json)

Useful, if the web server/service parses them.

cURL flags

cURL examples

curl -X GET -H "Accept: text/xml" http://localhost/agents/1◀Request XM

L Response

curl in action

with a Grails app

content detecting

service

REST

curl -X POST http://localhost/agents/?name=Ben

curl -X GET http://localhost/agents/1

curl -X PUT http://localhost/agents/1/?name=Benny

curl -X DELETE http://localhost/agents/1

◀Create

◀Read

◀Update

◀Delete

curl in action

with a Grails app

REST service

-d [variable]or--data [variable]Set a data block to transmit to the URL.Automatically sets the action to POST.

cURL flags

curl -d "agent[name]=Ann Banks" http://localhost/agents/1◀POST data with nam

e

curl passing a

data variable

-o [filename]Output results to a file.

All formatting and encoding intact.

cURL flags

http is just the beginning

Supported Protocols‣ HTTP‣ HTTPS‣ FTP‣ FTPS‣ TFTP‣ SCP‣ SFTP‣ Telnet‣ LDAP

The operations desk called.

They say our apps is using too much

network bandwidth.

Can you take a look at the transmitted

data?

What’s being sent over the wire?

Purpose‣ Output headers of packets from a network.

‣ Filter results with powerful expressions.

‣ Save captured data to a file.

TCPDump

Useful for‣ Debugging network traffic.

‣ Examining socket communications.

‣ Inspecting web service calls.

‣ Examining network data as a 3rd party

TCPDump

TCPDump commands

tcpdump -i en1

TCPDump commands

tcpdump -i lo0

TCPDump commands

tcpdump -i en1 -v◀I

ncrea

sed p

acket i

nfo

TCPDump commands

tcpdump -i en1 -n◀p

ort n

umbers, n

ot na

mes

TCPDump commands

tcpdump -i en1 -A◀A

SCII

out

put

TCPDump commands

tcpdump -i en1 -X◀H

EX +

ASC

II out

put

TCPDump commands

tcpdump -i en1 icmp

◀Exp

ressio

n

TCPDump commands

tcpdump -i en1 tcp port 80

◀Exp

ressio

n

Des

tinat

ion

Add

ress

Sour

ce A

ddre

ss

Leng

th

DSA

P

SSA

P

Con

trol

SNA

P

Dat

a

FCS

Ethernet Frame, up to 1514 bytes

TCPDump commands

tcpdump -i en1 -s 1514

◀ sn

aplen

gth

TCPDump commands

tcpdump -i en1 -c20◀C

aptu

re 20

packets

Wireshark (Cross Platform)

PurposeNetwork protocol analyzer.

A GUI on tcpdump

Wireshark

Useful featuresOnline and offline operation.Reading TCPDump data files.

Wireshark

Useful featuresLogical packet parsing, display.Decryption.

Gotcha on Mac OSXAt a terminal, upon each boot, type:sudo chown YOURUSERNAME /dev/bpf*

Gotcha on WindowsPromiscuous mode doesn’t always work.

Wireshark usage

Wireshark in action

Eavesdrop

Purpose-A Mac-specific network packet inspector.-Listens to TCP traffic between machines.

Eavesdrop (mac)

Eavesdrop in action

Why does our app crash whenever we

have a lot of users?

How many?

Oh, 10... or 1000.Something like that.

Maybe 10,000.

PurposeLoad test or sequence test HTML, EJBs, SOAP, and many other interfaces.

JMeter (Cross Platform)

Useful ForTesting web sites under strain.Finding load-induced problems.Graphing performance metrics.

JMeter (Cross Platform)

Caution!

Denial of Service

JMeter in action

JMeter in action

JMeter acting Groovy

Survey

StackOverflow.com

What’s your favoriteOpen Source Debugging Tool?

VisualVM

❸

Eclipse

❷

System.out.println(brokenobj);

❶

System.out.println(brokenobj);System.out.println(brokenobj);

System.out.println(brokenobj);

System.out.println(brokenobj);

System.out.println(brokenobj);

System.out.println(brokenobj);

System.out.println(brokenobj);

System.out.println(brokenobj);

System.out.println(brokenobj);

System.out.println(brokenobj);

System.out.println(brokenobj);

System.out.println(brokenobj);

System.out.println(brokenobj);

System.out.println(brokenobj);

buckle up

Java-Centric Tools

OSS Debug Tools

Web Service

JavaScriptFilesystem

Java

Network

JPS Process List

What JVM options did we use when we started up the app

server?

I’m gonna need a list of the other java processes running on

that box too.

jps

jps -l

◀Sho

w fu

ll pack

age na

me

jps -l -v

◀Sho

w fu

ll pack

age na

me

◀Sho

w JV

M arg

uments

jps -l -v -m

◀Sho

w fu

ll pack

age na

me

◀Sho

w JV

M arg

uments

◀Sho

w ma

in() a

rgume

nts

only local?

jps 10.15.25.32

How?

Run jstatd daemon

Allow-all policy

jstatd -J-Djava.security.policy=allowall.pol

◀You

r Poli

cy

grant codebase "file:${java.home}/../lib/tools.jar" { permission java.security.AllPermission;};

Another port besides 1099?

jstatd -J-Djava.security.policy=your.pol -p 1099

◀You

r Poli

cy◀P

ort

jps 10.15.25.32

jstat process info

jstat -options

jstat -gcutil <pid>

◀Sho

w gar

bage

colle

ction

jstat -gcutil <pid> 500 999◀m

s betw

een s

ample

s

◀repeti

tions

jstat -gcutil -h5 <pid> 500 999

◀Sho

w he

ader

every

X lin

es

jmap memory MAPs

Heap Dump

jmap -dump:file=myfile <pid>

Histogram

jmap -histo <pid>

jstack stack dumps

jstack <pid>

jhat Heap Analysis

jhat <heapfile>

jhat myheapdump.hprof

Javap Class File Disassembler

We added a new method to a class.

Is it on theserver-deployed copy

of the app?

javap classfile

javap -v classfile

◀Verb

ose

javap -v -s classfile

◀Verb

ose

◀Intern

al typ

e sign

atures

javap -v -s -l classfile

◀Verb

ose

◀Intern

al typ

e sign

atures

◀Disp

lay va

riable

table

s

javap -v -s -l -private classfile

◀Verb

ose

◀Intern

al typ

e sign

atures

◀Disp

lay va

riable

table

s

◀Sho

w Priva

te me

thod

s

We’re having, um some “problems” with the app in

production.

You’ll need to figure it out on the live production app.

No, you can’t restart the app!

No, you can’t put Eclipse on the

production box!!

PurposeDeeply inspect many metrics of Java processes.

Useful for‣ Discovering the GC cycles of your app.

Useful for‣ Discovering the GC cycles of your app.

‣ Finding the largest memory usage culprits.

Useful for‣ Discovering the GC cycles of your app.

‣ Finding the largest memory usage culprits.

‣ Diffing memory snapshots.

Useful for‣ Discovering the GC cycles of your app.

‣ Finding the largest memory usage culprits.

‣ Diffing memory snapshots.

‣ Injecting btrace code.

Useful for‣ Discovering the GC cycles of your app.

‣ Finding the largest memory usage culprits.

‣ Diffing memory snapshots.

‣ Injecting btrace code.

‣ Snapping heapdumps.

VisualVM in action

GotchaDownload version 1.1.1 even if your JDK includes 1.0.Standalone version always ahead of JVM version.

JStatd Daemon

We don’t have screens on our

servers.

Can you run these VisualVeeeEmmmms

remotely?

jstatd -J-Djava.security.policy=your.pol◀You

r Poli

cy

Remotely‣ Attach VisualVM

‣ Run any VisualVM debugging tool

‣ Run JPS

SerialVer Hash

Is the deployed version of the bean compatible with the copy we are

about to release?

Yeah, we don’t set serialVersionUID

serialver classname

MyBean: static final long serialVersionUID = -4193605393175618625L;

serialver -show

Eclipse Memory Analyzer(MAT)

Find out what objects are using so much memory in the

app!

Purpose‣ Visually investigate memory usage inside

a heap dump.

EclipseMemoryAnalyzer

PurposeInvestigate memory usage.

Useful forVisualizing memory footprint.

Finding memory leaks.

Querying through (OQL) allocated memory.

Discovering heavy memory allocation sources.

EclipseMemoryAnalyzer

GotchaDoesn’t create heap dumps, just analyzes.Available as a plugin or standalone RCP app.

EclipseMemoryAnalyzer

EclipseMemoryAnalyzer in action

ObjectQueryLanguage

EclipseMemoryAnalyzer in action

Retained Size Graphs

EclipseMemoryAnalyzer in action

Can you put a few debugging

“sysouts” into that running app not he

server?

OriginsDTrace on Sun SolarisPorted to other UNIX platforms

Now available as BTrace for Java

BTrace Tracing

Production-timeOperates on deployed codeRuntime injection of tracing statements

BTrace Tracing

Ken Sipe, contributing

CodeTracing written in Java.Doesn’t have to be compiled.

BTrace Tracing

ConsoleIt’s core strength.Original target.

BTrace Tracing

GUITightly integrated with VisualVM

BTrace Tracing

BTrace in action

Omniscient Debuggers

WebServiceCentric

OSS Debug Tools

Web Service

JavaScriptFilesystem

Java

Network

Purpose‣ Test, mock, and visualize web services.

SOAPUI

Useful for‣ Testing your app against a stub web

service.

‣ Testing a public web service via a GUI.

SOAPUI

SOAPUI in action

JavaScriptCentric

OSS Debug Tools

Web Service

JavaScriptFilesystem

Java

Network

PurposeVisually inspect an entire HTML DOM.

Useful forTesting HTML and CSS changes.Making realtime changes to the DOM.Monitoring network traffic.Debuging JavaScript.

FireBug

FireBug in action

FireBug in action

FireBug in action

FilesystemCentric

OSS Debug Tools

Web Service

JavaScriptFilesystem

Java

Network

fs_usage

PurposeShows file activity.

Useful forExamining which files are being written.Discovering what component is disk bound.

fs_usage (Mac, Linux)

fs_usage params

fs_usage -f filesys java◀M

ode/

info fil

ter◀C

omma

nd na

me

lsof

PurposeShows open files.

Useful forFinding what PID is holding a file open.

lsof (Mac, Linux)

lsof params

lsofFind all

open files

lsof params

lsof -p 4347

◀PID

flag

◀PID

numb

er

Find

everything

this PID has

open

Purpose #2Shows open network ports and PIDs.

Useful forFinding what PID is holding a port open.

lsof (Mac, Linux)

lsof params

Wait!

I thought

you said

files?

lsof params

lsof -iTCP:8080

◀Protoco

l◀P

ort

PurposeShows open files.Enumerates network TCP handles.Shows registry read/writes.

Useful forDiscovering network and disk activity causes.

ProcessMonitor (windows)

ProcessMonitor (windows)

An open source alternative

What Else?

Source Code search engines for examples

‣http://google.com/codesearch

‣ http://www.jexamples.com/

‣ http://www.codefetch.com/

‣ http://www.krugle.com/

‣ http://codesearch.developer.emc.com/

‣ http://www.codase.com/

‣ http://www.koders.com/

MatthewTwitter @matthewmccull

Blog http://www.ambientideas.com/blog sidebar has all my social media links

Email matthewm@ambientideas.com

GitHub http://github.com/matthewmccullough

CURLhttp://curl.haxx.se/

CygWinhttp://www.cygwin.com/

TCPDumphttp://www.tcpdump.org/

Wiresharkhttp://www.wireshark.org/

Eavesdrophttp://www.baurhome.net/software/eavesdrop/

Resources

Apache JMeterhttp://jakarta.apache.org/jmeter/

JMeter Groovy Pluginhttp://code.google.com/p/jmeter-groovy-sampler/

Resources

VisualVMhttps://visualvm.dev.java.net/

GCHistohttps://gchisto.dev.java.net/

Eclipse Memory Analyzerhttp://www.eclipse.org/mat/

BTracehttps://btrace.dev.java.net/

DTrace, XRay, Instrumentshttp://en.wikipedia.org/wiki/DTrace

Resources

SOAPuihttp://www.soapui.org/

Firebughttp://getfirebug.com/

fs_usagehttp://rentzsch.com/macosx/fs_usageIntro

lsofhttp://www.akadia.com/services/lsof_intro.html

Process Monitorhttp://technet.microsoft.com/en-us/sysinternals/bb896645.aspx

Resources

‣ http://www.ambientideasphotography.com‣ http://upload.wikimedia.org/wikipedia/commons/a/a4/

BernardMadoff.jpg‣ http://flickr.com/photos/triller/2226679393/‣ http://flickr.com/photos/ektogamat/2687444500/‣ http://flickr.com/photos/bfionline/2380398799/‣ http://flickr.com/photos/symphoney/76513801/‣ http://flickr.com/photos/foxypar4/2124673642/‣ http://flickr.com/photos/morberg/3146874095/‣ http://flickr.com/photos/triller/2226679393/

Image Credits