automatically generated patches as debugging aids
TRANSCRIPT
Automatically Generated Patches as Debugging Aids: A
Human Study
Yida Tao, Jindae Kim, Sunghun Kim
Dept. of CSE, The Hong Kong University of Science and Technology
Chang Xu
State Key Lab for Novel Software Technology, Nanjing University
• Promising research progress• ClearView1: Prevent all 10 Firefox
exploits• GenProg2: Fix 55/105 real bugs
[1] Automatically Patching Errors in Deployed Software. Perkins et al. SOSP’09[2] A systematic study of automated program repair: fixing 55 out of 105 bugs for $8 each. Le Goues et al. ICSE’122
Automatic Program Repair
3
Automatic Program Repair
4
- Slashdot discussion: http://science.slashdot.org/story/09/10/29/2248246/Fixing-Bugs-But-Bypassing-the-Source-Code
“It won't get your bug patched any quicker. You’ll just have shifted the coders' attention away from their own app's bugs, and onto the repair tool’s bugs.”
Automatic Program Repair
#what-could-possibly-go-wrong• Blackbox repair• Increasing maintenance
cost• Vulnerable to attack
- Slashdot discussion: http://science.slashdot.org/story/09/10/29/2248246/Fixing-Bugs-But-Bypassing-the-Source-Code- A human study of patch maintainability. ISSTA’12- Automatic patch generation learned from human-written patches. ICSE’13
5
- Slashdot discussion: http://science.slashdot.org/story/09/10/29/2248246/Fixing-Bugs-But-Bypassing-the-Source-Code- A human study of patch maintainability. ISSTA’12- Automatic patch generation learned from human-written patches. ICSE’13
#program-out-of-control
6
#what-could-possibly-go-wrong• Blackbox repair• Increasing maintenance
cost• Vulnerable to attack
7
Use automatically generated patches as debugging aids
8
Use automatically generated patches as debugging aids
Our Human Study• Investigate the usefulness of
generated patches as debugging aids
• Discuss the impact of patch quality on debugging performance
• Explore practitioners’ feedback on adopting automatic program repair
9
Methodology
10
Bugs
Participants
Debugging aid
Debugis given to
BugsParticipantsDebugging aid 11
BugsParticipantsDebugging aid 12
Low-quality generated patch
BugsParticipantsDebugging aid 13
Low-quality generated patch
High-quality generated patch
BugsParticipantsDebugging aid 14
Low-quality generated patch
High-quality generated patch
Buggy method location
BugsParticipantsDebugging aid 15
Grad: 44
Engr: 28
MTurk: 23
95 Participants
CS graduate students
Industrial software engineers
Amazon Mechanical Turk workers
BugsParticipantsDebugging aid 16
BugsParticipantsDebugging aid 17
44 Graduate students• Between-group
design
14 students
15 students
15 students
BugsParticipantsDebugging aid 18
44 Graduate students• Between-group
design
Low-quality generated patch
High-quality generated patch
Buggy method location
14 students
15 students
15 students
BugsParticipantsDebugging aid 19
44 Graduate students• Between-group
design• Onsite setting• Eclipse IDE• Supervised session
Low-quality generated patch
High-quality generated patch
Buggy method location
14 students
15 students
15 students
BugsParticipantsDebugging aid 20
Low-quality generated patch
High-quality generated patch
Buggy method location
Remote participants(28 Engr + 23 MTurk)• Within-group design
BugsParticipantsDebugging aid 21
Remote participants(28 Engr + 23 MTurk)• Within-group design• Online debugging system
Low-quality generated patch
High-quality generated patch
Buggy method location
BugsParticipantsDebugging aid 22
BugsParticipantsDebugging aid 23
Bug Selection Criteria
• Real bugs• The bug has accepted patches written by developers• Proper number of bugs• The bug has generated patches with different quality
BugsParticipantsDebugging aid 24
Automatic patch generation learned from human-written patches. Kim et al. ICSE’13
BugsParticipantsDebugging aid 25
Automatic patch generation learned from human-written patches. Kim et al. ICSE’13
Auto-generated patch A Auto-generated patch B
for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) if(sub!=null){ args[i+1] = sub.toString(); }}
for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) args[parenCount+1] = new Integer(reImpl.leftContext.length);}
BugsParticipantsDebugging aid 26
Automatic patch generation learned from human-written patches. Kim et al. ICSE’13
Auto-generated patch A Auto-generated patch B
avg. ranking from 85 devs and students
for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) if(sub!=null){ args[i+1] = sub.toString(); }}
for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) args[parenCount+1] = new Integer(reImpl.leftContext.length);}
1.6
2.8
BugsParticipantsDebugging aid 27
Automatic patch generation learned from human-written patches. Kim et al. ICSE’13
Auto-generated patch A Auto-generated patch B
avg. ranking from 85 devs and students
High-Quality Patch Low-Quality patch
for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) if(sub!=null){ args[i+1] = sub.toString(); }}
for (int i=0; i<parenCount; i++) SubString sub = (SubString)parens.get(i) args[parenCount+1] = new Integer(reImpl.leftContext.length);}
1.6
2.8
BugsParticipantsDebugging aid 28
BugsParticipantsDebugging aid 29
Participants submit 337 patches as their debugging outcome
BugsParticipantsDebugging aid 30
Participants submit 337 patches as their debugging outcome
Location109
LowQ112
HighQ116# submitted
patches w.r.t debugging aid
BugsParticipantsDebugging aid 31
Participants submit 337 patches as their debugging outcome
Location109
LowQ112
HighQ116# submitted
patches w.r.t debugging aid
Bug166
Bug274
Bug359
Bug476
Bug562
# submitted patches w.r.t bugs
32
Evaluation of debugging performance
Patch CorrectnessCorrectness
33
Patch Correctness
• Passing test casesCorrectness
34
Patch Correctness
• Passing test cases• Matching the semantics of original
accepted patches
Correctness
35
Patch Correctness
• Passing test cases• Matching the semantics of original
accepted patches• 3 evaluators
Correctness
36
Debugging Time
• Eclipse Plug-in•Website Timer
Correctness
Debugging time
37
Correctness
Debugging time
• Independent variables• Debugging aids• Bugs• Participant types• Programming
experience
38
Multiple Regression Analysis
Correctness
Debugging time
• Independent variables• Debugging aids• Bugs• Participant types• Programming
experience
correctness=α0+α 1 ∙ x1+α 2 ∙ x2+α 3 ∙ x 3+α 4 ∙ x4d ebugging time=β0+β1 ∙ x1+β2 ∙ x2+β3 ∙ x3+β 4 ∙ x4
39
Post-study Survey
• Helpfulness of debugging aids• Difficulty of bugs• Opinions on using generated patches as
debugging aids
Correctness
Debugging time
Survey feedback
40
41
Results
42
High-quality patches significantly improve debugging correctness
1
48%
33%
71%
43
High-quality patches significantly improve debugging correctness
1
48%
33%
71%
Lo-catio
n
LowQ HighQ
% of correct patches
48%
71%
44
Lo-catio
n
LowQ HighQ
% of correct patches
High-quality patches significantly improve debugging correctness
1
Positive Coefficient =
1.25p-value= 0.00 < 0.05
48%
71%
45
Lo-catio
n
LowQ HighQ
% of correct patches
Low-quality patches slightly undermine debugging correctness
2
48%
33%
71%
46
Lo-catio
n
LowQ HighQ
% of correct patches
Low-quality patches slightly undermine debugging correctness
2
Negative Coefficient = -0.55p-value= 0.09 48%
33%
71%
47
Lo-catio
n
LowQ HighQ
% of correct patches
Low-quality patches can undermine debugging correctness
2
Negative Coefficient = -0.55p-value= 0.09 48%
33%
71%
48
High-quality patches are more useful for difficult bugs3
49
High-quality patches are more useful for difficult bugs3
2
3
4
5
Bug Difficulty
Bug1Math-280
Bug2Rhino-114493
Bug3Rhino-192226
Bug4Rhino-217379
Bug5Rhino-76683
50
High-quality patches are more useful for difficult bugs3
Bug1 Bug2 Bug3 Bug4 Bug50%
10%
20%
30%
40%
50%
60%
70%
80%
90%
% of correct patches
Location LowQ HighQ
2
3
4
5
Bug Difficulty
Bug1Math-280
Bug2Rhino-114493
Bug3Rhino-192226
Bug4Rhino-217379
Bug5Rhino-76683
51
4The type of debugging aid does not affect debugging time
52
4The type of debugging aid does not affect debugging time
Location LowQ HighQ0
20
40
Debugging time (min)
53
5Other factors’ impact on debugging performance
Difficult bugs significantly slow down debuggingEngr and MTurk are more likely to debug correctly Novices tend to benefit more from HighQ patches
Helpfulness of de-bugging aidsVery
helpful
Helpful
Medium
Slightly Helpful
Not Helpful
54
Participants consider high-quality generated patches much more helpful than low-quality patches
Low-quality generated patch
High-quality generated patch
Mann-Whitney U testp-value =
0.001
6
55
Feedback
56
57
Quick starting point• Point to the buggy
area• Brainstorm
“They would seem to be useful in helping find various ideas around fixing the issue, even if the patch isn’t always correct on its own.”
58
Quick starting point• Point to the buggy
area• Brainstorm
Confusing, incomplete, misleading• Wrong lead, especially for
novices• Require further human
perfection“They would seem to be useful in helping find various ideas around fixing the issue, even if the patch isn’t always correct on its own.”
59
“Generated patches would be good at recognizing obvious problems”
“…but may not recognize more involved defects.”
60
“Generated patches would be good at recognizing obvious problems”
“…but may not recognize more involved defects.”
“Generated patches simplify the problem”
“…but they may over-simplify it by not addressing the root cause.”
61
“I would use generated patches as debugging aids, as they provide extra diagnostic information”
62
“I would use generated patches as debugging aids, as they provide extra diagnostic information”
“…along with access to standard debugging tools.”
63
Threats to Validity
64
Threats to Validity
• Bugs and generated patches may not be representative• Quality measure of generated patches may not
generalize• May not generalize to domain experts• Possibility of blindly reusing generated patches• Remove patches that are submitted less than 1 minute
65
Takeaway
• Auto-generated patches can be useful as debugging aids• Participants fix bugs more correctly
with auto-generated patches
• Quality control is required• Participants’ debugging correctness is
compromised with low-quality generated patches
• Maximize the benefits• Difficult bugs• Novice developers