Download - Test ROI v4 General
The ROI of TestingThe ROI of Testing
Presented By:Shaun BradshawQuestcon Technologies
The ROI of TestingThe ROI of Testing
Presented By:Shaun BradshawQuestcon Technologies
Quality - Innovation - VisionQuality - Innovation - Vision
Slide Slide 22
Return on InvestmentReturn on Investment
Slide Slide 33
Return on Investment (ROI)Return on Investment (ROI)
• What is ROI?ROI is a calculation that attempts to determine
the actual or perceived future value of an
expense or investment
By calculating ROI, an organization can assess
whether the expense/investment is justified by
the resulting savings/revenue
Slide Slide 44
Return on Investment (ROI)Return on Investment (ROI)
• How is ROI calculated?In its most basic form ROI is calculated as
follows:
ROI = (Benefits – Costs)/Costs
That is, the financial benefit after an investment
or improvement is made minus the cost of the
investment or improvement, calculated as a
percentage of those costs Benefits• Revenue generated• Cost reduction• Cost avoidance• Productivity
improvements
Costs• Labor expense• Time expense• Tool expense
Slide Slide 55
• What makes testing valuable? How do we get to a positive ROI?
Produces information reliably grounded in
observed system behavior [1]
– Functions like a credit check [1] – Exposes risk and improves decision-making
Extends the life of the software
Improves the development process
Increases end-user confidence and satisfaction
Return on Investment (ROI)Return on Investment (ROI)
Slide Slide 66
Defect Preventionvs.
Defect Detection
Defect Preventionvs.
Defect Detection
Slide Slide 77
• Importance of TestingOrganizations rely on testers to do QA
We cannot “test” quality into an application
Test needs to be the “backdoor to QA”
Requires a shift in the testing paradigm
Test Finding
Defects
Defect Prevention vs. Defect Detection Defect Prevention vs. Defect Detection
Measuring Quality
Slide Slide 88
• Rules & Results of Defect Prevention
60% Rule [2]
Percentage of defects
introduced in require-
ments and design
85% Rule [2]
Percentage of defects
removed through
reviews, inspections
and testing
368:1 Rule [3]
Defect Prevention vs. Defect Detection Defect Prevention vs. Defect Detection
Slide Slide 99
Design Code Test ProductionRequirements
30
30
$0
$0
0
30
60
0
$0
$0
30
90
0
$0
$0
10
100
85
$166,600
$166,600
0
15
$378,700
15
$212,100
# of defects introducedAccumulated defects
Defects removed
Cost
Accumulated Cost
X 5X 10X 50X 368X+
Testing Only
Assume X is $100 &
100 total defects
Defect Prevention vs. Defect Detection Defect Prevention vs. Defect Detection
Test Only Costs
0
50000
100000
150000
200000
250000
300000
Require
ments
Test Only Costs
0
50000
100000
150000
200000
250000
300000
Require
ments
Design
Test Only Costs
0
50000
100000
150000
200000
250000
300000
Require
ments
Design
Code
Test Only Costs
0
50000
100000
150000
200000
250000
300000
Require
ments
Design
CodeTes
t
Test Only Costs
0
50000
100000
150000
200000
250000
300000
Require
ments
Design
CodeTes
t
Produc
tion
Slide Slide 1010
Slide Slide 1111
Quantitative BenefitsQuantitative Benefits
Slide Slide 1212
• Reduced Defect Repair CostsTest involvement earlier in the lifecycle reduces repair costsAutomation improves defect detectionGood test practices increase detection likelihood
Quantitative Benefits Quantitative Benefits
Full QATest OnlyAd Hoc Testing
$0
$100,000
$200,000
$300,000
$400,000
$500,000
$600,000
Slide Slide 1313
• Decreased Production CostsDefects in production can be significantly higher than 368x -
the “Annuity Nightmare”– Small rounding error goes undetected at a major financial institution– Months until discovered– Code repair is cheap– Production cost – NOT CHEAP!
• Increased Revenue & ProfitsReduced time-to-market for functioning software
Increased market share
Higher customer retention/goodwill
Reduced maintenance costs (easier to enhance)
Quantitative Benefits Quantitative Benefits
Slide Slide 1414
Calculating Test ValueCalculating Test Value
Slide Slide 1515
• Defect Injection RateTotal number of defects introduced into an application
When each defect was introduced (using Root Cause
Analysis)
Number of defects introduced in each phase of the SDLC
Calculate the defect injection rate as:
# of defects introduced
total defects
Key Metrics to Calculate Test Value Key Metrics to Calculate Test Value
Slide Slide 1616
• Defect Repair CostHourly rate of each resource involved in the repair (PM, BA,
Developer, Tester, etc.)
Defect repair time for each resource by phase of the SDLC
Number of defects repaired in each phase of the SDLC
Calculate the defect repair costs as shown in the example*:
*Data in the example was drawn from [4].
Key Metrics to Calculate Test Value Key Metrics to Calculate Test Value
Slide Slide 1717
• “Test” EffectivenessNumber of defects found during “testing” activities
during any phase of the SDLC
Total number of defects found including in production
(use a “warranty period”)
Calculate test effectiveness as:
# of defects foundphase
total # of defects found
Key Metrics to Calculate Test Value Key Metrics to Calculate Test Value
Slide Slide 1818
ROI ExampleROI Example
Slide Slide 1919
ROI Example ROI Example
Slide Slide 2020
Established a test team made up of:• 1 test lead• 1 test analyst• 1 test consultant
Total 1st year investment: $237,500
Established a test team made up of:• 1 test lead• 1 test analyst• 1 test consultant
Total 1st year investment: $237,500
ROI Case Study ROI Case Study
Slide Slide 2121
Conducted 2 projects in first 8 monthsRealized $173,000 savings in project-over-project defect repair costsCalculated savings do not include other benefits such as:
•Reduced production downtime•Reduced maintenance costs• Increased customer satisfaction & trust
Anticipated 46% ROI at the end of year 1
Conducted 2 projects in first 8 monthsRealized $173,000 savings in project-over-project defect repair costsCalculated savings do not include other benefits such as:
•Reduced production downtime•Reduced maintenance costs• Increased customer satisfaction & trust
Anticipated 46% ROI at the end of year 1
ROI Case Study ROI Case Study
Slide Slide 2323
References References
1. Bullock, James. “Calculating the Value of Testing.” Software Testing & Quality Engineering: May/June, 2000. <www.stickyminds.com/>
2. Jones, Capers. "Software Cost Estimating Methods for Large Projects." CrossTalk: April, 2005. <www.stsc.hill.af.mil/crosstalk/2005/04/0504Jones.html>
3. Dabney, JB. "Return on Investment of Independent Verification and Validation Study Preliminary Phase 2B Report." Fairmont, WV: NASA IV&V Facility, 2003. <sarpresults.ivv.nasa.gov/ViewResearch/289/24.jsp>
4. US Dept. of Commerce, National Institute of Standards & Technology (NIST). “Planning Report 02-3: The Economic Impacts of Inadequate Infrastructure for Software Testing.” Technology Program Office, Strategic Planning & Economic Analysis Group. May, 2002.<www.nist.gov/director/prog-ofc/report02-3.pdf>