value-based software test prioritization

University of Southern California

Center for Systems and Software Engineering

Value-Based Software Test Prioritization

Annual Research ReviewCSSE-USC

Qi Li, Barry Boehm{qli1, boehm}@usc.edu

March 7, 2012



Outline

• Research Motivation• Research Method• Case Studies• Tool Support• Conclusion

04/19/2023 ARR 2012 2



Research Motivation• Value-neutral SE methods are increasingly risky [Boehm, 2003]

– Every requirement, use case, object, test case, and defect is equally important– “Earned Value” Systems don’t track business value– System value-domain problems are the chief sources of software project failures

• Testing & Inspection resources are expensive and scarce– 30%-50%, even higher for high reliability projects [Ramler, 2005]– Time-to-market [Boehm, Huang, 2005]

• Empirical Findings [Bullock 2000, Boehm & Basili 2001 ]– About 20 percent of the features provide 80 percent of business value– About 80 percent of the defects come from 20 percent of the modules– …

• Value-based Software Engineering 4+1 theorem [Boehm, 2005]

04/19/2023 ARR 2012 3



Outline


04/19/2023 ARR 2012 4



Value-Based Software Test Prioritization

What to be prioritized? •Testing items: Testing Scenarios, Testing Features, Test Cases

How to prioritize?•Value-Based (Business Importance, Risk, Cost)•Dependency Aware

How to Measure?•Average Percentage of Business Importance Earned (APBIE)



Research Method: Value-Based• Risk Exposure (RE)

– Where Size (Loss) is the risk impact size of loss if the outcome is unsatisfactory, Pro (Loss) is the probability of an unsatisfactory outcome

• Risk Reduction Leverage (RRL)

– Where REbefore is the RE before initiating the risk reduction effort and REafter is the RE afterwards.

– RRL is a measure of the cost-benefit ratio of performing a candidate risk reduction or defect removal activity

604/19/2023 ARR 2012



Research Method: Value-Based

– Business Case Analysis– Stakeholder

Prioritization

– Impact of Defect

– Experience Base

04/19/2023 ARR 2012 7

– Business Value

Defect Criticality

– Defect-prone Components, Performers

• Value-Based Prioritization Drivers:

– Size of Loss

– Probability of Loss

Risk Exposure

• Testing items are to be ranked by how well they can reduce RE



Research Method: Value-Based• Combining with the testing items’ relative costs• =>Priority Trigger:

• This proposed strategy enables them to be prioritized in terms of Risk Reduction Leverage (RRL) or ROI

• Supposed to improve the lifecycle cost-effectiveness of defect removal techniques

04/19/2023 ARR 2012 8



Animated displays

Secondary application functions

User amenities

Main application functions

Natural speech input

Tertairy application functions

Basic application functions

Data management system

Operating System

Investment High-payoff Diminishing returns

Cost of software product [Boehm, 1981]Va

lue

of

so

ftw

are

pro

du

ct

to o

rga

niz

ati

on

Research Method: Dependency Aware

04/19/2023 9ARR 2012



Research Method: Dependency Aware• Dependency:

– Example: dependencies among test cases to be executed

– Solution: Prioritization Algorithm (greedy alg)• Select the one with the

highest RRL• Check dependency

04/19/2023 ARR 2012 10

9->3->9->5->9->4->7



Research Method: Metrics• Testing Cost Effectiveness

– Average Percentage of Business Importance Earned (APBIE)

04/19/2023 11ARR 2012



Outline


04/19/2023 ARR 2012 12



Case Studies Results• Exercise Test Prioritization based on Risk

Reduction Level (RRL)software testing scenarios to be walked through in

Galorath.Incsoftware features to be tested in a Chinese companysoftware test cases to be executed in USC SE course

projects

All of them show preliminary positive results

04/19/2023 ARR 2012 13



Case Studies Results (Galorath Inc.)Prioritize testing scenarios to be walked through• Galorath Inc. (2011 Summer)

04/19/2023 ARR 2012 14

APBIE-1 70.99%

APBIE-2 10.08%

APBIE-3 32.10%

– Value-based prioritization can improve the cost-effectiveness of testing

8 10 12 14 16 18 20 22 24 26 28 30

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

35.80%

39.51%

45.68%

51.85%

58.02%61.73%

87.65%

4.94%6.17%

9.88%

16.05%22.22%

25.93%

58.02%

74.07%

77.78%

83.95%

90.12% 93.83%95.06%

100.00%

PBIE-1

PBIE-2

PBIE-3

Stop Testing

Value-based

Value-neutral

Value-inverse (worst case)



Outline


04/19/2023 ARR 2012 15



Automated Tool Support (Beta Version)

04/19/2023 ARR 2012 16



Tool Demo

04/19/2023 ARR 2012 17

Website: http://greenbay.usc.edu/dacs/vbt/testlink/index.php

http://greenbay.usc.edu/dacs/vbt/testlink/index.php



Future Features• Establish the traceability matrix between the requirement

specifications and test cases to automatically obtain test case business importance ratings

• Establish the traceability matrix between test cases and defects in order to automatically predict the fail probability based on the collected historical defect data via incorporating the-state-of-art defect prediction techniques

• Experiment sensitivity analysis for reasoning and judging the correctness of factors’ ratings, or how ratings changes will impact the testing order

04/19/2023 ARR 2012 18

If you would like to be a beta version tester, please contact me at [email protected]



Outline


04/19/2023 ARR 2012 19



Conclusion• Propose a Real “Earned Value” System to Track Business Value

of Testing and Measure Testing Efficiency in terms of APBIE• Propose a Systematic Strategy for Value-based, Dependency

Aware Test Processes• Apply This Strategy to a Series of Empirical Studies with

different granularities of Prioritizations• Elaborate Decision Criteria of Testing Priorities Per Project

Contexts, Which are Helpful for Real Industry Practices• Implement an automatic tool for its application on large-scale

industrial projects

04/19/2023 ARR 2012 20



Question and Answer

04/19/2023 ARR 2012 21

value-based software test prioritization

Documents

software testing

software engineeringthis

software engineeringcurrent

software quality

research plan

software defect reduction

software development

research approach