icse 2011 ds_1
TRANSCRIPT
![Page 1: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/1.jpg)
1
Pragmatic Prioritization of Software Quality Assurance Efforts
Emad ShihabQueen’s University
![Page 2: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/2.jpg)
2
Motivation
Software ^ has bugsalways
always
How to allocate quality assurance efforts for legacy and new code?
Q:
managers ^ have limited resources
![Page 3: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/3.jpg)
3
One Answer: Defect Prediction
Many recent studies focused on defect prediction
….However adoption in practice remains low
![Page 4: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/4.jpg)
4
An Industrial Example
How to prioritize the creation of unit tests for a large legacy system
![Page 5: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/5.jpg)
5
Criteria for Pragmatic QA Approaches
Focused Granularity
Unit of prioritization must be focused (e.g., function or method)
Current Approach Desired Approach
File, Folder or subsystem level (Thousands of LOC)
![Page 6: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/6.jpg)
6
Criteria for Pragmatic QA Approaches
Responsive Feedback
Continuous reflection of system changes.
Current Approach Desired Approach
Predict for next release
![Page 7: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/7.jpg)
7
Criteria for Pragmatic QA Approaches
Consider Effort
Consider effort in recommendations (e.g., use size)
Current Approach Desired Approach
No consideration for effort
In addition, need to evaluate the generality of the approaches to understand implications of findings
![Page 8: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/8.jpg)
8
Proposed Pragmatic SQA Approaches
Unit test creation
How to allocate quality assurance efforts for legacy code?Q:
A:
Change risk
How to allocate quality assurance efforts for new code?
Q:
A:
Complete
In Progress
![Page 9: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/9.jpg)
9
Unit Test CreationApproach
1. Search modification record comments for keywords, bug identifiers and modified files
2. Extract source code of modified file(s) and compare to previous version to identify changed functions
3. Combine data from 1 and 2 to calculate function-level heuristics
![Page 10: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/10.jpg)
10
Unit Test CreationHeuristics
Most Frequently Modified (MFM)Most Recently Modified (MRM)
Most Frequently Fixed (MFF)Most Recently Fixed (MRF)
Largest Fixed (LF)Largest Modified (LM)
Change Risk (CR)Size Risk (SR)
Random
Modification
Fix
Size
RiskRandom
![Page 11: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/11.jpg)
11
Unit Test CreationHeuristic Performance
Based on a heuristic, generate list of 10 functions to write unit tests for
Use size of function to measure effort required to write unit test
![Page 12: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/12.jpg)
12
Unit Test Creation Usefulness: Was writing the unit test useful?
Time to write unit testA
B
C
6 bug fixes
2 bug fixes
0 bug fixes
Usefulness = 2/3 = 66.67%
![Page 13: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/13.jpg)
13
Unit Test CreationUsefulness of Heuristics
Random
Most Recently Modified (MRM)
Size Risk (SR)
Change Risk (CR)
Most Recently Fixed (MRF)
Most Frequently Modified (MFM)
Most Frequently Fixed (MFF)
Largest Modified (LM)
Largest Fixed (LF)
0 10 20 30 40 50 60 70 80 90 100
27.7
43.1
48.8
55
56.9
80
83.8
84.7
87
8.5
9.9
12.6
17.4
16
28.1
32.3
32.9
44.7
Open SourceCommercial
![Page 14: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/14.jpg)
Revisiting the Pragmatic QA CriteriaUnit Test Creation
Unit Test CreationFunction level recommendations
Heuristic lists are continuously updated
Size of function used to consider effort required
Focused Granularity
Consider Effort
Responsive Feedback
Evaluate Generality: Studied Open Source and Commercial systems
![Page 15: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/15.jpg)
15
Change RiskProposed Approach
1. Mine changes and extract change metrics
2. Identify bug-inducing changes
3. Predict risky changes
![Page 16: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/16.jpg)
Revisiting the Pragmatic QA CriteriaChange Risk
Change RiskMap changes to modified functions
Flag risky changes while they are fresh in developer’s minds
Size of change used to consider effort
Focused Granularity
Consider Effort
Responsive Feedback
Evaluate Generality: Study on Open Source and Commercial systems
![Page 17: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/17.jpg)
Thank youI am grateful for this opportunity!
![Page 18: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/18.jpg)
Backups
![Page 19: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/19.jpg)
Revisiting the Pragmatic SQA Criteria
Unit Test Creation Change Risk
Focused Granularity
Estimate Effort
Evaluate Generality
Timely Feedback
![Page 20: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/20.jpg)
Pragmatic SQA Criteria
What we have What we wantFile level Unit of prioritization
must be focused (e.g., function)
Act on results in future release
Must be able to act on results of approach quickly
No effort provided Provide an estimate of the effort required for the task
Focused Granularity
Estimate Effort
Timely Feedback
Evaluate Generality
To help understand implications of findings
![Page 21: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/21.jpg)
21
Changes RiskNew code
Identify risky changes for closer examination before they are integrated into the code base
![Page 22: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/22.jpg)
22
Unit Test CreationImplemented code
Prioritize the creation of unit tests for implemented code
![Page 23: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/23.jpg)
23
Revisiting the Pragmatic-aware Criteria
Focused Granularity Timely feedback Estimate Effort Evaluate Generality
![Page 24: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/24.jpg)
24
Avoid the most bugs effectively!
Write unit tests for functions with best Return on Investment (ROI)
How can we avoid the most bugs given limited resources?
![Page 25: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/25.jpg)
25
Defect Prediction... An Example
Uses code metrics and pre-release defects to predict files with post-release defects in the Eclipse project.
![Page 26: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/26.jpg)
26
Motivation
Software has^ bugs and
managers have ^ limited resources
always
always
How to allocate quality assurance efforts for implemented and new code?
Q:
A: Re-active and pro-active SQA
![Page 27: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/27.jpg)
27
Proposed ApproachExtracting Historical Data
1. Extract change data to calculate metrics (e.g., churn)2. Use SZZ to identify and predict bug-inducing changes3.
![Page 28: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/28.jpg)
28
Answer … Defect Prediction!
Use code metrics and process metrics to predict which files are most likely to contain defects in future releases
![Page 29: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/29.jpg)
29
Defect Prediction... The Response
“The results are interesting, but of limited use to us ...”
... Why?
![Page 30: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/30.jpg)
30
Defect Prediction... Lessons Learned
1. Identifying files is too coarse of a granularity
2. Having to wait till the next release is too late
3. Need an estimate of effort required
4. Why should we believe this will work for us?
![Page 31: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/31.jpg)
31
Criteria for Pragmatic SQA Approaches
Timely Feedback
Must be able to act on results of approach quickly
![Page 32: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/32.jpg)
32
Criteria for Pragmatic SQA Approaches
Estimate Effort
Provide an estimate of the effort required for the task
![Page 33: Icse 2011 ds_1](https://reader035.vdocument.in/reader035/viewer/2022081512/58f0f6851a28ab6f408b45a9/html5/thumbnails/33.jpg)
33
Criteria for Pragmatic SQA Approaches
Evaluate Generality
Evaluate on multiple projects/domains to clearly understand implications of the findings