Public Deployment of Cooperative Bug Isolation

Ben Liblit, Mayur Naik, Alice Zheng, Alex Aiken, and Michael Jordan

UC Berkeley, Stanford University and University of Wisconsin (pending)

Our Goal: Measure Reality

• We measure bridges, airplanes, cars…– Where is ight data recorder for software?

• Users are a vast, untapped resource– 96,000 new Kazaa users during this workshop– Users know what matters most

• Opportunity for reality-directed debugging– Implicit bug triage for an imperfect world

Bug Isolation Architecture

ProgramSource

Compiler

Sampler

Guesses

ShippingApplication

Pro le& /

€ƒƒ

€‚Statistical

Debugging

Top bugs withlikely causes

Predicates on Program Behavior

• Guess what might be interesting– Branches: Left? Right?– Function returns: Negative? Zero? Positive?– Pairs of variables: Less? Equal? Greater?– Reference counts: Alive? Dead?

• Count how often guesses are true

• Feedback: vector of counts + outcome label

Sampling the Bernoulli Way

• “Next sample” countdown– Geometric distribution

• Split into acyclic regions– Finite threshold weight

Sampling the Bernoulli Way

• “Next sample” countdown– Geometric distribution

• Split into acyclic regions– Finite threshold weight

1

2 1

1

1

2

3

4

Sampling the Bernoulli Way

• “Next sample” countdown– Geometric distribution

• Split into acyclic regions– Finite threshold weight

• Clone acyclic regions– “Fast” & “slow” variants

– Choose at run time

• Result:– Subset of dynamic behavior

– Statistically fair sample

>4?

Multithreaded Programs

• Global next-sample countdown– High contention, small footprint– Want to use registers for performanceThread-local: one countdown per thread

• Global predicate counters– Low contention, large footprintOptimistic atomic increment

Multi-Module Programs

• Forget about global static analysis– Plug-ins, shared libraries– Instrumented & uninstrumented code

• Self-management at compile time– Locally derive identifying object signature– Embed static site information within object le

• Self-management at run time– Report feedback state on normal object unload– Signal handlers walk global object registry

Native Compiler Integration

• Instrumentor must mimic native compiler– You don’t have time to port & annotate by hand

• Our approach: source-to-source, then native

• Hooks for GCC:– Stage wrapping via scripts– Flag management via spec les

ProgramSource

Compiler

Sampler

Guesses

ShippingApplication

CompilerTool Chain

Keeping the User In Control

Public Deployment, To Date

0

200

400

600

800

1000

1200

1400

Evolu

tion

Gaim

GIM

P

Gnum

eric

Nautil

us

Rhythm

box

Rep

orts

Rec

eive

d

GoodErrorCrash

Public Deployment, To Date

0%

20%

40%

60%

80%

100%

Evolu

tion

Gaim

GIM

P

Gnum

eric

Nautil

us

Rhythm

box

GoodErrorCrash

Sneak Peak: Data Exploration

Conclusions

• Public deployment is challenging– Real world code pushes tools to their limits– Large user communities take time to build

• But the results are worth it:

“Thanks to Ben Liblit and the CooperativeBug Isolation Project, this version of

Rhythmbox should be the most stable yet.”

The Cooperative Bug Isolation Projecthttp://www.cs.berkeley.edu/~liblit/sampler/

Join the Cause!