mutation-aware fault predictioncrest.cs.ucl.ac.uk/cow/52/slides/cow52_hall.pdf · 2017-04-03 ·...
TRANSCRIPT
![Page 1: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/1.jpg)
MUTATION-AWARE
FAULT PREDICTIONDavid Bowes: Hertfordshire University, UK
Tracy Hall: Brunel University London, UK
Mark Harman: University College London, UK
Yue Jia: University College London, UK
Federica Sarro: University College London, UK
Fan Wu: University College London, UK
![Page 2: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/2.jpg)
Agenda
• Introduction
• Background
• Methods
• Results
• Conclusions
![Page 3: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/3.jpg)
Introduction
• Predicting defects is important but hard.
![Page 4: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/4.jpg)
Background: Defect prediction
• ….
4
Data about the code
Data analysis method
Defective code
predictions
![Page 5: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/5.jpg)
Background: mutation testing
• Creates a mutant of a program.
• A mutant is `killed' (identified) by a test case.
• Mutants are created by mutation operators
![Page 6: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/6.jpg)
Methods: Mutation Operators
![Page 7: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/7.jpg)
Research Questions
RQ1: Is the performance of predictive modelling techniques
improved when they have mutation metrics available?
RQ2: What is the effect size of improvements in predictive
model performance due to mutation metrics?
RQ3: What is the relative performance of static and
dynamic mutation metrics?
![Page 8: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/8.jpg)
Methods• Subject systems…
![Page 9: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/9.jpg)
Methods: Subject Systems
![Page 10: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/10.jpg)
Methods: Data collected
• Defect data collected using SZZ Algorithm
• Commonly used source code metrics collected…
![Page 11: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/11.jpg)
Methods: Metrics
![Page 12: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/12.jpg)
Methods: Data collected
• Defect data collected using SZZ Algorithm
• Commonly used source code metrics collected using
JHawk.
• PITest used to mutate the code and identify methods
where the unit tests do not detect/kill the mutation
• Our tool then computes five types of
Static and dynamic mutation metrics
![Page 13: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/13.jpg)
Methods: Types of Mutation Metrics
![Page 14: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/14.jpg)
Methods: Data collected
• Defect data collected using SZZ Algorithm
• Commonly used source code metrics collected using
JHawk.
• PITest used to mutate the code and identify methods
where the unit tests do not detect/kill the mutation
• Our tool then computes five types of
Static and dynamic mutation metrics
• PITEST has 7 mutations operators
• We used 40 mutation metrics in total
![Page 15: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/15.jpg)
Methods: Machine Learning Techniques
• Techniques used to predict if a class is defective:
• Naïve Bayes
• Logistic Regression
• J48
• Random Forest
• Used the WEKA Wrapper Subset Selection Filter.
• Performed 10-fold cross validation repeated 50 times
• Evaluated the results using Matthews Correlation
Coefficient
• MCC values range between -1 and +1.
![Page 16: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/16.jpg)
Results
RQ1: Is the performance of predictive modelling techniques
improved when they have mutation metrics available?
RQ2: What is the effect size of improvements in predictive
model performance due to mutation metrics?
RQ3: What is the relative performance of static and
dynamic mutation metrics?
![Page 17: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/17.jpg)
RQ1: Do Mutation Metrics Improve Performance?
C- code
S- static
D- dynamic
A- All
![Page 18: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/18.jpg)
RQ1: Do Mutation Metrics Improve Performance?
![Page 19: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/19.jpg)
RQ1: Do Mutation Metrics Improve Performance?
Apache
![Page 20: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/20.jpg)
RQ2: What is the effect size of improvements in
predictive model performance due to mutation metrics?
Top 10 most frequently used metrics
![Page 21: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/21.jpg)
RQ2: What is the effect size of improvements in
predictive model performance due to mutation
metrics?Frequency of mutation metrics in the top 10 metrics
![Page 22: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/22.jpg)
RQ2: What is the effect size of improvements in
predictive model performance due to mutation metrics?
Effect of each metric to model performance: global
![Page 23: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/23.jpg)
RQ2: What is the effect size of improvements in
predictive model performance due to mutation metrics?
Effect of each metric to model performance: local
![Page 24: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/24.jpg)
RQ3: What is the relative performance of static and
dynamic mutation metrics?
C- code
S- static
D- dynamic
A- All
![Page 25: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/25.jpg)
RQ3: What is the relative performance of
static and dynamic mutation metrics?
![Page 26: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/26.jpg)
Conclusions
• Mutation awareness can significantly improve predictive
performance and with worthwhile effect sizes.
• The effects vary from:
• System to system
• Algorithm to algorithm
• Combinations of metrics are generally good.
• Our results are good for:
• Defect prediction
• Mutation testing
• More work needed….
![Page 27: MUTATION-AWARE FAULT PREDICTIONcrest.cs.ucl.ac.uk/cow/52/slides/cow52_Hall.pdf · 2017-04-03 · MUTATION-AWARE FAULT PREDICTION David Bowes: Hertfordshire University, UK Tracy Hall:](https://reader033.vdocument.in/reader033/viewer/2022052721/5f0b8f247e708231d4311c98/html5/thumbnails/27.jpg)
Questions?
Highlights:
• Our work addresses important gaps in defect prediction:• Testing Information rarely used
• Dynamic Information rarely used
• Industrial data rarely used
• Findings have lots of promise:• Adding mutation information improves predictive
performance
• Models are significantly improved
• Worthwhile effect sizes occur
• Could improve mutation testing?
• A novel way of working out the effect of metrics