predict software inspection effectivesess j.k. orr 07-09-2015

Predicting Software Inspection Process Defect Detection

James K. OrrIndependent Consultant

[email protected]

Copyright by James K. Orr 2015

Contents

• Goal• Extreme Care With Data • A Software Metric Story• Development Of Data

– Ideal Inspection Team– Inspection Team Effectiveness Prediction Process– Real World Inspection Teams, No Knowledge Of Capability– Real World Inspection Teams, Inspector Capability Known

• Observations • Candidate Data Collection Format


Goal


Be Able To Accurate Compute Long Term Team Inspection Effectiveness

Based On Data Available At Completion Of Required Number Of

Samples Of Inspections


Extreme Care With Data


Warning On Potential For Misuse

• This presentation attempts to show significant advantages in use of data as collected on the Space Shuttle Primary Avionics Software (PASS) project for quality and process management.

• However, this data must never be used in personnel decisions such as appraisals.

• In the hands of software process professionals, it can provide accurate assessment of the state of the project, manage inspection quality, and information for team or individual training.


A Software Metric Story


A Software Metric Story• Software Metrics Saleman

“I have a great new metric to sell you. With this metric, you can immediately know how many defects your project has missed in software inspections!”

• Senior Manager“I hear many software metric stories before. Give me some proof.”

• Software Metrics Saleman“Let me have your data in the following format, and I will give you the answer.”

• Senior Manager“I will have my metrics person start collecting the data, this is not something we normally collect. Call me in six months.”


A Software Metric Story

Six Months Later• Project Metric Analyst

“We have collected the data you requested. Here is your data.”• Software Metrics Saleman

“I will have you an answer in an hour!”

Six Months, Plus 1 Hour Later• Software Metrics Saleman

“You Found 132 major defects in 40 inspections. But, you missed 123 major defects.”

• Senior Manager“Thanks for the input. Check back with me in three years and I will let you know how good your estimate is.”


Behind The Scenes

• Data Provided


Behind The Scenes

• Analysis


Estimated Inspection Effectiveness


Estimated Vs Actual Defects Missed


Development Of Data


Ideal Inspection Teams


Establish Data For Ideal Inspection

• My research indicated much smaller variation if all inspectors had identical characteristics– The probability that each inspector would find

each major defect was exactly the same• Simulations were run with four inspectors

inspecting a product with six major defects.– Simulated 1,000 times• Establish average for all 1,000 times



• This was repeated for 1,000 cases, 6 Defects Present– 1,000 averages of 1,000 attempts (1,000,000 total)

• Establishes statistic characteristics of ideal inspection

– Use Capture-Recapture Metric (know at end of inspection)

Major Defects Found By Exactly Two Inspectors---------------------------------------------------------------

Major Defects Found By Exactly One Inspector Only

– Predicts Team Inspection Effectiveness –

• Percent All Defects Found During Inspection Divided By All Defects In Product (Known and Unknown)


Team Inspection Effectiveness Vs Capture-Recapture Metric


Key Observations

• For Ideal Inspections (those with all inspectors having the same probability of detecting each errors)– Defects Found By Exactly Two Inspectors Divided

By Defects Found By Exactly One Inspector predicts • Percent All Defects Found During Inspection Divided By

All Defects In Product (Known and Unknown)


Simulation Results, Each Point Is The Average Of 1000 Simulation


All Inspectors Have 30 % Probability Of Finding

Major Errors




Major Errors

Inspection Team EffectivenessPrediction Process


Prediction Process

• Gather Data Which Includes Information On Each Inspector’s Detection Of Each Error

• For The Above Example– 2 Defects Discovered By Exactly One Inspector– 1 Defect Discovered By Exactly Two Inspectors– Overall Inspection Effectiveness = 50 % (3 of 6 Major Defects Detected By Team)


Defect 1 Defect 2 Defect 3 Defect 4 Defect 5 Defect 6

Inspector 1 0 1 0 1 0 0

Inspector 2 0 0 0 1 0 0

Inspector 3 0 0 1 0 0 0

Inspector 4 0 0 0 0 0 0

• Compute Capture-Recapture Metric Major Defects Found By Exactly Two Inspectors---------------------------------------------------------------

Major Defects Found By Exactly One Inspector Only

• Compute Capture-Recapture Metric = ½– Value 50 %

• Determine Team Inspection Effectiveness Value From average of Ideal Inspection– See Next Chart– Value 68 %


Team Inspection Effectiveness Vs Capture-Recapture Metric


Capture-Recapture

Metric

Raw Estimate Of Inspection Team

EffectivenessOne Data Point


• My research indicated much larger variation if for realistic variation for inspectors – Previously computed individual inspector probability of

major error detection (using aliases, not real names) for Space Shuttle Primary Avionics Software project.

• Simulations were run with four inspectors inspecting a product with six major defects.– Simulated 1,000 times

• Establish average for all 1,000 times• Also, one of 1,000 times was selected as random result


Real World Inspection TeamsNo Knowledge Of Capability

Use Data From Inspections As They Occur


Approach

• Minimum data is collected from each inspection as it occurs

• Multiple Inspections Are Grouped Together To Compute Recapture Metric (5, 10, 20, etc.)

• For each composite set of Inspections, estimate is made of Project Inspection Effectiveness



Illustration Of MethodCharacteristics Of One

Random Inspection TeamResults Of 1 Simulation With

Fixed Inspection Team

Metric For Raw Estimated Inspection Effectiveness, Sample

Size = 5

Relationship For “Ideal Inspection Team”

Repeat Process For Different Random Inspection Team

Process Capability Based On 1,000 Simulations

Estimated Inspection EffectivenessIdeal Team, All 20 % Probability


Project C Inspector Distribution


Estimated Inspection EffectivenessProject C


Project B Inspector Distribution


Estimated Inspection EffectivenessProject B


Project A Inspector Distribution


Estimated Inspection EffectivenessProject A


Real World Inspection Teams Inspector Capability Known


Uncontrolled Environment

• The difficulty of using real project data to predict true inspection team effectiveness is that virtually nothing is constant over time.

• Inspection Variables– Number of Inspectors– New members to organization (new hire inspectors)– Training / Growth of Individual’s Capability– Number of Defects In Inspected Material– Different difficulty of finding different types of errors


Overview Of Process


Knowledge Of Inspector Capability

Pre-Assign Inspectors To

Specific Teams

Simulate Expected Average Performance

Of Each Team

Combine All Inspections For Project

Potential Solution

• The process I call “statistical simulation” offers a promise to control the environment.

• “Statistical simulation” is equivalent to rolling a pair of dice 10,000 times to see what the expected outcome is.

• With real world inspector distributions, the data becomes much more varied.

• Data variation (standard deviation) can become a variable that allows for calibration of results based on Ideal Inspection data.


Test Inspection Team Profiles

• Four arbitrary (but somewhat realistic) organization inspector profiles were created– Vary the differences between poorest and best

inspectors (as measured by probability an inspector will detect a major error)

– Estimate Inspection Team Effectiveness using different sample sizes, and determine standard deviation which will be larger with greater differences between inspector capability.

– Hopefully, the resulting actual Inspection Team Effectiveness for the four profiles will be similar.


Project A Inspector Distribution


Illustration Of Method


Characteristics Of One Random Inspection Team

Results Of 1,000 Simulation With Fixed Inspection Team

Actual Effectiveness Higher Than For Ideal Team For Same Recapture Metric

Relationship For “Ideal Inspection Team”

Metric For Raw Estimated Inspection Effectiveness, Sample

Size = 5

Repeat Process For Different Random Inspection Team

Simulation Result Project A


Estimated Inspection EffectivenessProject A


Project B Inspector Distribution


Simulation Result Project B


Estimated Inspection EffectivenessProject B


Project C Inspector Distribution


Simulation Result Project C


Estimated Inspection EffectivenessProject C


Adjustment Required If Team Composition Is Know (Simulation)


Observations


Method

• This presentation was put together with data from a very simple simulation, with no attempt to provide refinements such as shown on page 53 (Adjustment Required If Team Composition Is Know (Simulation))

• A refined model could be produced for a specific project if historical data on similar prior project existed.

• Contact me for more information.


Real World Inspection Teams, No Knowledge Of Capability

• Data was shown for four distributions Of Inspector Capability (probability to find major defect in inspection material) with overall average of each inspector finding 20 % of defects

• Pattern of estimated Inspection Effectiveness with sample size was similar for all four distributions

• A similar process done for a real world project’s inspector distribution could establish confidence limits for analyzing sets of inspections

• However, the variance is very large


Real World Inspection Teams, Inspector Capability Known

• Knowledge Of Inspector Characteristics allows for prediction of an arbitrary inspection team’s Inspection Effectiveness capability with small variance

• Data supporting determination of Inspector Characteristics should be maintained with alias identification of individual inspectors and never used by management for appraisals– Results have a large random component– Some inspectors may be in multiple inspection with no or

very few defects, hence no opportunity to find errors in inspections


Real World Inspection Teams, Inspector Capability Known

• Knowledge Of Inspector Characteristics allows for pre-evaluation of an arbitrary inspection team to see if project standards for minimum Inspection Effectiveness.– Additional inspectors could be added prior to the

inspection if needed to minimum Inspection Effectiveness• Collection of additional data such as defect type

could allow in depth process / team capability leading to identification of needed training at the project level or individual inspector level


Candidate Data Collection Format


Inspection Data

•


Related Defect Data


predict software inspection effectivesess j.k. orr 07-09-2015

Documents

development of data

inspections copyright

goal copyright

times copyright

scenes data

data available

software inspections

use of data