automatically generated patches as debugging aids

Automatically Generated Patches as Debugging Aids: A

Human Study

Yida Tao, Jindae Kim, Sunghun Kim

Dept. of CSE, The Hong Kong University of Science and Technology

Chang Xu

State Key Lab for Novel Software Technology, Nanjing University

• Promising research progress• ClearView1: Prevent all 10 Firefox

exploits• GenProg2: Fix 55/105 real bugs

[1] Automatically Patching Errors in Deployed Software. Perkins et al. SOSP’09[2] A systematic study of automated program repair: fixing 55 out of 105 bugs for $8 each. Le Goues et al. ICSE’122

Automatic Program Repair

3

Automatic Program Repair

4

- Slashdot discussion: http://science.slashdot.org/story/09/10/29/2248246/Fixing-Bugs-But-Bypassing-the-Source-Code

“It won't get your bug patched any quicker. You’ll just have shifted the coders' attention away from their own app's bugs, and onto the repair tool’s bugs.”

Automatic Program Repair

#what-could-possibly-go-wrong• Blackbox repair• Increasing maintenance

cost• Vulnerable to attack

- Slashdot discussion: http://science.slashdot.org/story/09/10/29/2248246/Fixing-Bugs-But-Bypassing-the-Source-Code- A human study of patch maintainability. ISSTA’12- Automatic patch generation learned from human-written patches. ICSE’13

5

- Slashdot discussion: http://science.slashdot.org/story/09/10/29/2248246/Fixing-Bugs-But-Bypassing-the-Source-Code- A human study of patch maintainability. ISSTA’12- Automatic patch generation learned from human-written patches. ICSE’13

#program-out-of-control

6

#what-could-possibly-go-wrong• Blackbox repair• Increasing maintenance

cost• Vulnerable to attack

7

Use automatically generated patches as debugging aids

8

Use automatically generated patches as debugging aids

Our Human Study• Investigate the usefulness of

generated patches as debugging aids

• Discuss the impact of patch quality on debugging performance

• Explore practitioners’ feedback on adopting automatic program repair

9

Methodology

10

Bugs

Participants

Debugging aid

Debugis given to

BugsParticipantsDebugging aid 11

BugsParticipantsDebugging aid 12

Low-quality generated patch

BugsParticipantsDebugging aid 13

Low-quality generated patch

High-quality generated patch

BugsParticipantsDebugging aid 14

Low-quality generated patch

High-quality generated patch

Buggy method location

BugsParticipantsDebugging aid 15

Grad: 44

Engr: 28

MTurk: 23

95 Participants

CS graduate students

Industrial software engineers

Amazon Mechanical Turk workers

BugsParticipantsDebugging aid 16

BugsParticipantsDebugging aid 17

44 Graduate students• Between-group

design

14 students

15 students

15 students

BugsParticipantsDebugging aid 18

44 Graduate students• Between-group

design

Low-quality generated patch

High-quality generated patch

Buggy method location

14 students

15 students

15 students

BugsParticipantsDebugging aid 19

44 Graduate students• Between-group

design• Onsite setting• Eclipse IDE• Supervised session

Low-quality generated patch

High-quality generated patch

Buggy method location

14 students

15 students

15 students

BugsParticipantsDebugging aid 20

Low-quality generated patch

High-quality generated patch

Buggy method location

Remote participants(28 Engr + 23 MTurk)• Within-group design

BugsParticipantsDebugging aid 21

Remote participants(28 Engr + 23 MTurk)• Within-group design• Online debugging system

Low-quality generated patch

High-quality generated patch

Buggy method location

BugsParticipantsDebugging aid 22

BugsParticipantsDebugging aid 23

Bug Selection Criteria

• Real bugs• The bug has accepted patches written by developers• Proper number of bugs• The bug has generated patches with different quality

BugsParticipantsDebugging aid 24

Automatic patch generation learned from human-written patches. Kim et al. ICSE’13

BugsParticipantsDebugging aid 25

Automatic patch generation learned from human-written patches. Kim et al. ICSE’13

Auto-generated patch A Auto-generated patch B

for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) if(sub!=null){ args[i+1] = sub.toString(); }}

for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) args[parenCount+1] = new Integer(reImpl.leftContext.length);}

BugsParticipantsDebugging aid 26

Automatic patch generation learned from human-written patches. Kim et al. ICSE’13

Auto-generated patch A Auto-generated patch B

avg. ranking from 85 devs and students

for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) if(sub!=null){ args[i+1] = sub.toString(); }}

for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) args[parenCount+1] = new Integer(reImpl.leftContext.length);}

1.6

2.8

BugsParticipantsDebugging aid 27

Automatic patch generation learned from human-written patches. Kim et al. ICSE’13

Auto-generated patch A Auto-generated patch B

avg. ranking from 85 devs and students

High-Quality Patch Low-Quality patch

for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) if(sub!=null){ args[i+1] = sub.toString(); }}

for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) args[parenCount+1] = new Integer(reImpl.leftContext.length);}

1.6

2.8

BugsParticipantsDebugging aid 28

BugsParticipantsDebugging aid 29

Participants submit 337 patches as their debugging outcome

BugsParticipantsDebugging aid 30

Participants submit 337 patches as their debugging outcome

Location109

LowQ112

HighQ116# submitted

patches w.r.t debugging aid

BugsParticipantsDebugging aid 31

Participants submit 337 patches as their debugging outcome

Location109

LowQ112

HighQ116# submitted

patches w.r.t debugging aid

Bug166

Bug274

Bug359

Bug476

Bug562

# submitted patches w.r.t bugs

32

Evaluation of debugging performance

Patch CorrectnessCorrectness

33

Patch Correctness

• Passing test casesCorrectness

34

Patch Correctness

• Passing test cases• Matching the semantics of original

accepted patches

Correctness

35

Patch Correctness

• Passing test cases• Matching the semantics of original

accepted patches• 3 evaluators

Correctness

36

Debugging Time

• Eclipse Plug-in•Website Timer

Correctness

Debugging time

37

Correctness

Debugging time

• Independent variables• Debugging aids• Bugs• Participant types• Programming

experience

38

Multiple Regression Analysis

Correctness

Debugging time

• Independent variables• Debugging aids• Bugs• Participant types• Programming

experience

correctness=α0+α 1 ∙ x1+α 2 ∙ x2+α 3 ∙ x 3+α 4 ∙ x4d ebugging time=β0+β1 ∙ x1+β2 ∙ x2+β3 ∙ x3+β 4 ∙ x4

39

Post-study Survey

• Helpfulness of debugging aids• Difficulty of bugs• Opinions on using generated patches as

debugging aids

Correctness

Debugging time

Survey feedback

40

41

Results

42

High-quality patches significantly improve debugging correctness

1

48%

33%

71%

43

High-quality patches significantly improve debugging correctness

1

48%

33%

71%

Lo-catio

n

LowQ HighQ

% of correct patches

48%

71%

44

Lo-catio

n

LowQ HighQ

% of correct patches

High-quality patches significantly improve debugging correctness

1

Positive Coefficient =

1.25p-value= 0.00 < 0.05

48%

71%

45

Lo-catio

n

LowQ HighQ

% of correct patches

Low-quality patches slightly undermine debugging correctness

2

48%

33%

71%

46

Lo-catio

n

LowQ HighQ

% of correct patches

Low-quality patches slightly undermine debugging correctness

2

Negative Coefficient = -0.55p-value= 0.09 48%

33%

71%

47

Lo-catio

n

LowQ HighQ

% of correct patches

Low-quality patches can undermine debugging correctness

2

Negative Coefficient = -0.55p-value= 0.09 48%

33%

71%

48

High-quality patches are more useful for difficult bugs3

49

High-quality patches are more useful for difficult bugs3

2

3

4

5

Bug Difficulty

Bug1Math-280

Bug2Rhino-114493

Bug3Rhino-192226

Bug4Rhino-217379

Bug5Rhino-76683

50

High-quality patches are more useful for difficult bugs3

Bug1 Bug2 Bug3 Bug4 Bug50%

10%

20%

30%

40%

50%

60%

70%

80%

90%

% of correct patches

Location LowQ HighQ

2

3

4

5

Bug Difficulty

Bug1Math-280

Bug2Rhino-114493

Bug3Rhino-192226

Bug4Rhino-217379

Bug5Rhino-76683

51

4The type of debugging aid does not affect debugging time

52

4The type of debugging aid does not affect debugging time

Location LowQ HighQ0

20

40

Debugging time (min)

53

5Other factors’ impact on debugging performance

Difficult bugs significantly slow down debuggingEngr and MTurk are more likely to debug correctly Novices tend to benefit more from HighQ patches

Helpfulness of de-bugging aidsVery

helpful

Helpful

Medium

Slightly Helpful

Not Helpful

54

Participants consider high-quality generated patches much more helpful than low-quality patches

Low-quality generated patch

High-quality generated patch

Mann-Whitney U testp-value =

0.001

6

55

Feedback

56

57

Quick starting point• Point to the buggy

area• Brainstorm

“They would seem to be useful in helping find various ideas around fixing the issue, even if the patch isn’t always correct on its own.”

58

Quick starting point• Point to the buggy

area• Brainstorm

Confusing, incomplete, misleading• Wrong lead, especially for

novices• Require further human

perfection“They would seem to be useful in helping find various ideas around fixing the issue, even if the patch isn’t always correct on its own.”

59

“Generated patches would be good at recognizing obvious problems”

“…but may not recognize more involved defects.”

60

“Generated patches would be good at recognizing obvious problems”

“…but may not recognize more involved defects.”

“Generated patches simplify the problem”

“…but they may over-simplify it by not addressing the root cause.”

61

“I would use generated patches as debugging aids, as they provide extra diagnostic information”

62

“I would use generated patches as debugging aids, as they provide extra diagnostic information”

“…along with access to standard debugging tools.”

63

Threats to Validity

64

Threats to Validity

• Bugs and generated patches may not be representative• Quality measure of generated patches may not

generalize• May not generalize to domain experts• Possibility of blindly reusing generated patches• Remove patches that are submitted less than 1 minute

65

Takeaway

• Auto-generated patches can be useful as debugging aids• Participants fix bugs more correctly

with auto-generated patches

• Quality control is required• Participants’ debugging correctness is

compromised with low-quality generated patches

• Maximize the benefits• Difficult bugs• Novice developers