elaine weyuker august 2014. to determine which files of a large software system with multiple...

Elaine WeyukerAugust 2014

To determine which files of a large software system with multiple releases are likely to contain the largest numbers of bugs in the next release.

Help testers prioritize testing efforts.

Help developers decide when to do design and code reviews and what to re-implement.

Help managers allocate resources.

Verified that bugs were non-uniformly distributed among files.

Identified properties that correlated most closely with fault-proneness, and then built a statistical model and ultimately a tool to make predictions.

Size of file (KLOCs)Number of changes to the file in the previous 2 releases.

Number of bugs in the file in the last release.

Age of file (Number of releases in the system)

Language the file is written in.

All of the systems we’ve studied to date used a configuration management system which integrates version control and change management functionality, including bug history.

Initially manually read Modification Requests (MRs), decide which were bugs, and what sorts of changes, if any, were made to files.

Once we had a tool, data was automatically extracted and passed to the prediction engine.

Used Negative Binomial Regression

Also considered machine learning algorithms including:◦ Recursive Partitioning◦ Random Forests◦ BART (Bayesian Additive Regression Trees)

Consists of two parts.

The back end extracts data needed to make the predictions.

The front end makes the predictions and displays them.

Extracts necessary data from the repository.

Predicts how many bugs will be in each file in the next release of the system.

Sorts to files in decreasing order of the number of predicted bugs.

Displays results to user.

Percentage of actual bugs that occurred in the N% of the files predicted to have the largest number of bugs. (N=20)

Considered other measures less sensitive to the specific value of N.

System Years Followed

Releases LOC % Faults Top 20%

NP 4 17 538K 83%

WN 2 9 438K 83%

VT 2.25 9 329K 75%

TS 9+ 35 442K 81%

TW 9+ 35 384K 93%

TE 7 27 327K 76%

IC 4 18 1520K 91%

AR 4 18 281K 87%

IN 4 18 2116K 93%

What Are We Missing?

Fault Prediction Tool Overview

Prediction Engine

Statistical Analysis

Version Mgmt /Fault Database

(previous releases)

Release to be predicted

User-supplied parameters

Fault-proneness predictions

User enters system name.

User specifies that all problems reported in System Test phase are faults.

User asks for fault predictions for release “Bluestone2008.1”

Available releases are found in the version mgmt database. User chooses the releases to analyze.

User selects 4 file types.

User confirms configuration

User enters filename to save the configuration.

User clicks Save & Run button, to start the prediction process.

Initial prediction view for Bluestone2008.1

All files are listed in decreasing order of predicted faults

Listing is restricted to eC files

Listing is restricted to 10% of eC files

Prediction tool is fully-operational◦ 750 lines Python◦ 2150 lines C, 75K bytes compiled

Current version’s backend is specific for the internal AT&T configuration management system but can be adapted to other configuration management systems. All that is needed is a source of the data required by the prediction model.

Developers◦ Counts◦ Individuals

Calling StructureAmount of Code Change