an automated approach to predict effectiveness of fault localization tools tien-duy b. le, and david...
TRANSCRIPT
![Page 1: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/1.jpg)
An Automated Approach to Predict Effectiveness of Fault Localization Tools
Tien-Duy B. Le, and David LoSchool of Information Systems
Singapore Management University
Will Fault Localization Work For These Failures ?
29th IEEE International Conference on Software Maintenance
![Page 2: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/2.jpg)
Fault Localization Tool: A Primer
Give me a failing program
I have calculated the most suspicious location of bugs
Running
…
1)
2)
3)
4)
OK! I will check your suggestion
Debugging
My program failed
![Page 3: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/3.jpg)
3
Will Fault Localization Tools Really Work?
• In ideal case:– Faulty statements are within a few suspicious
statements, e.g. 10, 20, 30 …
…
1)
2)
3)
4)
I found the bugDebugging
Effectiv
e
![Page 4: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/4.jpg)
4
Will Fault Localization Tools Really Work?
• In the worst case:– Faulty statements cannot be found early in
the ranked list of statements– Time consuming
…
1)
2)
3)
4)
Debugging Forever Debugging
Effectiv
e
![Page 5: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/5.jpg)
5
Will Fault Localization Tools Really Work?
• We build an oracle to predict if an output of a fault localization tool (i.e., instance) can be trusted or not.
• If not trusted– Developers do not have to spend time using the output– Developers can revert to manual debugging
Trusted or not ?
Oracle Ball
![Page 6: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/6.jpg)
6
Overall Framework
Suspiciousness Scores
Spectra
Fault Localization
Feature Extraction
Model Learning1 2
Effectiveness Labels
Training Stage
Model
![Page 7: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/7.jpg)
7
Overall Framework
• Major Components:– Feature Extraction
• 50 features, 6 categories– Model Learning
• We extend Support Vector Machine (SVM) to handle imbalanced training data.
![Page 8: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/8.jpg)
8
Feature Extraction
Traces (5 Features)T1 # traces
T2 # failing traces
T3 # successful traces
…
Program Elements (10 Features)PE1 # program elements in failing traces
PE2 # program elements in correct traces
PE3 PE2 – PE1
…
![Page 9: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/9.jpg)
9
Feature Extraction
Raw Scores (10 Features)R1 Highest suspiciousness scores
R2 Second highest suspiciousness scores
Ri ith highest suspiciousness scores
…
Simple Statistics (6 Features)SS1 Number of distinct scores in the top-10 scores
SS2 Mean of top-10 suspiciousness scores
SS3 Median of top-10 suspiciousness scores
…
![Page 10: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/10.jpg)
10
Feature ExtractionGaps (11 Features)
G1 R1 – R2
G2 R2 – R3
Gi Ri – R(i+1), where
…
Relative Difference (8 Features)RD1
RDi101
102
RR
RR
)82( where,101
101 iRR
RRi
)93( i
![Page 11: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/11.jpg)
Model Learning• Extend off-the-shell Support Vector Machine• Imbalanced training data
– #ineffective instances > #effective instancesExtended Support Vector Machine (SVMEXT)
11
Maximum Marginal Hyperplane
Effective instances
Ineffective instances
![Page 12: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/12.jpg)
12
SVMEXT
• For each effective instance, – We calculate its similarities to ineffective instances– Each instance is represented by a feature vector– Using cosine similarity:
)11(
)()(
)(50
1
250
1
2
50
1
p
ba
bap
i ii i
i ii
![Page 13: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/13.jpg)
13
SVMEXT
• Sort effective instances based on their highest similarities with ineffective instances (descending)
• Duplicate effective instances at the top of the list until training data is balanced.
selected effective instances
Effective instances
Ineffective instances
![Page 14: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/14.jpg)
Overall Framework
14
Model
Suspiciousness Scores
Spectra
Fault Localization
Feature Extraction
Effectiveness Prediction
Prediction
Deployment Stage
1 3
![Page 15: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/15.jpg)
15
Experiments
RecallPrecision
RecallPrecision2Measure-F
Recall
Precision
FNTP
TPFPTP
TP
.
• We use 10 fold cross validation.• We compute precision, recall and F-measure.
![Page 16: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/16.jpg)
16
Effectiveness Labeling
• A fault localization instance is deemed effective if:– Root cause is among the top-10 most
suspicious program elements– If a root cause spans more than 1 program
elements• One of them is in the top-10
– Otherwise, it is ineffective
![Page 17: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/17.jpg)
17
Dataset
• 10 different programs: – NanoXML, XML-Security, and Space– 7 programs from the Siemens test suites
• Totally, 200 faulty versions• For Tarantula, among the 200 instances:
– 85 are effective– 115 are ineffective
![Page 18: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/18.jpg)
18
Research Question 1
• How effective is our approach in predicting the effectiveness of a state-of-the-art spectrum-based fault localization tool ?
• Experimental setting:– Tarantula – Using Extended SVM (SVMEXT)
![Page 19: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/19.jpg)
19
Research Question 1
• Precision of 54.36%– Correctly identify 47 out of 115 ineffective fault
localization instances• Recall of 95.29%
– Correctly identify 81 out of 85 effective fault localization instances
Precision Recall F-Measure54.36% 95.29% 69.23%
![Page 20: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/20.jpg)
20
Research Question 2
• How effective is our extended Support Vector Machine(SVMExt) compared with off-the-shelf Support Vector Machine (SVM) ?
• Experimental Setting– Tarantula – Using extended SVM (SVMEXT) and off-the-
shelf SVM
![Page 21: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/21.jpg)
21
Research Question 2
• Result
SVMEXT outperforms off-the-shelf SVM
SVMEXT SVM ImprovementPrecision 54.36% 51.04% 6.50%Recall 95.29% 57.65% 65.29%F-Measure 69.23% 54.14% 27.87%
![Page 22: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/22.jpg)
22
Research Question 3
• What are the most important features ?– Fisher score is used to measure how
dominant and discriminative a feature is.
![Page 23: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/23.jpg)
23
Top-10 Most Discriminative Features
50%
30%
10%
10%Relative Differences (5)
Program Elements (3)
Raw Scores (1)
Simple Statistics (1)
Gaps (0)
Traces (0)
1. RD72. RD83. RD64. PE15. PE26. SS17. RD58. RD19. PE410.R1
![Page 24: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/24.jpg)
24
Most Important Features
• Relative Differences Features – C7
(1), C8(2), C6
(3), C5(7), and C1
(8)
score) nesssuspicioushighest i :(R
)82( where,
thi
101
101
101
1021
iRR
RRC
RR
RRC
ii
![Page 25: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/25.jpg)
25
Most Important Features
• Program Elements– PE1
(4), PE2(5), and PE4
(9)
Failing Traces Correct Traces
#Program Elements PE1 PE2
2
14
PE
PEPE
![Page 26: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/26.jpg)
26
Most Important Features
• Simple Statistics– SS1(6): Number of distinct suspiciousness
scores in {R1,…,R10}
• Raw Scores– R1
(10): Highest suspiciousness scores
![Page 27: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/27.jpg)
27
Research Question 4
• Could our approach be used to predict the effectiveness of different types of spectrum-based fault localization tool ?
• Experimental setting:– Tarantula, Ochiai, and Information Gain– Using Extended SVM (SVMEXT)
![Page 28: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/28.jpg)
28
Research Question 4
Tool Precision Recall F-Measure
Tarantula 54.36% 95.29% 69.23%
Ochiai 63.23% 97.03% 76.56%
Information Gain 64.47% 93.33% 76.26%
• F-Measure for Ochiai and Information Gain– Greater than 75%– Our approach can better predict the effectiveness of
Ochiai and Information Gain
![Page 29: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/29.jpg)
29
Research Question 5
• How sensitive is our approach to the amount of training data ?
• Experimental setting:– Vary amount of training data from 10% to 90%– Random sampling
![Page 30: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/30.jpg)
30
Research Question 5
10% 20% 30% 40% 50% 60% 70% 80% 90%0%
20%
40%
60%
80%
100%
120%
Precision Recall F-Measure
Amount of Training Data
![Page 31: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/31.jpg)
31
Conclusion
• We build an oracle to predict the effectiveness of fault localization tools.– Propose 50 features capturing interesting
dimensions from traces and susp. scores– Propose Extend. Support Vector Machine (SVMEXT)
• Experiments– Achieve good F-Measure: 69.23% (Tarantula)– SVMEXT outperforms off-the-shelf SVM– Relative difference features are the best features
![Page 32: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/32.jpg)
32
Future work
• Improve F-Measure further• Extend approach to work for other fault
localization techniques• Extract more features from source code
and textual descriptions, e.g., bug reports.
![Page 33: An Automated Approach to Predict Effectiveness of Fault Localization Tools Tien-Duy B. Le, and David Lo School of Information Systems Singapore Management](https://reader037.vdocument.in/reader037/viewer/2022110403/56649e5d5503460f94b5542f/html5/thumbnails/33.jpg)
33
Thank you!
Questions? Comments? Advice?{btdle.2012, davidlo}@smu.edu.sg