1

Bug Isolation viaRemote Program Sampling

Ben Liblit Alex Aiken

Alice X. Zheng Michael Jordan

Presented By :

Arpita Gandhi

2

Motivation All deployed software systems have bugs

Resources for improvement are limited

The user community outnumbers the testing team

Current systems are not systematic and automated

3

Related Applications Commercial databases produce log files

and inspect them when the user reports a problem

Netscape/Mozilla, Microsoft, GNOME,KDE have opt-in crash reporting systems

Problems?

4

Goals Gather information systematically from

user executions

Perform automatic analysis to fix software quality problems using this information

Speed up the process of bug isolation

Develop an efficient, low overhead system

5

Discussion

What do you think about the idea of using users as debuggers?

Bugzilla showed 36,937 bugs and

an additional 60,191 bugs as duplicates of already reported bugs

6

Design Goals Modest impact on program performance

Statistically fair and uniformly random sampling

Efficient transmission of client data

7

Sampling Blocks

Consider the following piece of code{

check(p != NULL);p = p->next;check(i < max);total += sizes[i];

}We want to sample 1/100th of these checks

8

Sampling Blocks Solution 1 : Maintain a global counter

modulo 100 Problem?

for(i = 0 ; i < 10; i++) { check(p != NULL);

p = p->next; check(i < max); total += sizes[i]; }

9

Sampling Blocks Solution 2: Use a random number generator{ if(rnd(100) == 0)check (p != NULL); p = p->next; if(rnd(100) == 0) check (i < max); total += sizes[i];} Problem?

10

Sampling Blocks Solution 3: Anticipate future samples Requires two versions of code:

Slow path: Code with the sampled

instrumentation

Fast path: Code w/o the sampled

instrumentation

11

Sampling Blocks

Fast Path Code Slow Path Code

if (countdown > 2) {p = p->next;total += sizes[i];

}

if( countdown-- == 0 ) {check(p != NULL);countdown = getNextCountdown();}p = p->next;if( countdown-- == 0 ) {check( i < max );countdown = getNextCountdown();}total += sizes[i];

12

Sampling Functions Represent sampling blocks as a CFG Weight of path is the maximum number

of instrumentation sites Place a countdown threshold check on

each acyclic region For each region r:

If (next-sample countdown >weight) no samples taken

13

Instrumented Code Layout

>4?

14

Optimizations Problem: A new threshold check before

each function call

Solution: Identify and ignore weightless functions No threshold checks, instrumented

code Compiled without any modification

15

Optimizations Problem: Use of global variable for

countdown management expensive

Solution: Use Local Variables

Local 1

Local 2

Local 3

global

Function entry

Use for threshold check, sampling decisions, decrements

Function exit

16

Issues in Program Sampling Factors that hamper performance

Remote monitoring Local storage Use of network bandwidth Storage on central server For smaller level deployment? Adaptive refinement of

instrumentation

17

Other Issues Social and ethical concern

Users’ interest in contribution

Learning curve with software

More testers vs. remote sampling?

18

Applications Sharing cost of assertions

Isolating deterministic bugs

Isolating non – deterministic bugs

19

Experiments Sharing the cost of assertions

Sample assert ( ) statements in CCURED code

Overhead incurred:

Scenario Average Maximum

Unconditional 55% 181%

1/100 sampling

17% 46%

1/1000 sampling

10% 26%

20

Effectiveness of Sampling At density 1/1000 for observing rare

program behavior? To achieve confidence level =90%, Least number of runs needed = 230,258 ! Solution: No. of licensed Office XP users = 16 million

2 runs/week/user = 230258 runs every 19 min!

21

References Sampling User Executions for Bug

Isolation

Public Deployment of Cooperative Bug Isolation

Scalable Statistical Bug Isolation

www.cs.wisc.edu/liblit

22

Thank You