1
Applications of Optimization to Logic Testing
Gary Kaminski and Paul Ammann
ICST 2010
CSTVA Workshop
2
OutlineMotivation
Test Sets and Coverage CriteriaShould We Apply Well Known Optimization Techniques?
Testing DNF Predicates For Logic FaultsMinimal DNF SummaryDNF Faults and Fault HierarchyCoverage Criteria
Optimizing DNF Criteria
ResultsOptimized Test Set Sizes
Benefit, Computational CostOptimizing Individual Tests
Implications For Regression Testing
Thoughts On Where to Go from Here
3
MotivationTest Sets and Coverage Criteria
Problem 1: Find Test Set that Satisfies CriterionVery Common ApproachOften Tests Are Redundant
Problem 2: Find Minimal Test Set that Satisfies CriterionGiven A Test Set, Throw Out Redundant TestsNow Every Test Is There For a Reason!
Problem 3: Find Minimum Test Set that Satisfies CriterionGenerally Much HarderCan’t Be Found With “Local” InformationNeed To Make Globally Optimal Choices
Question:Is It Useful To Recast Coverage As Standard Optimization Problem?
Gain?Cost?
Here We Address This Question In Context Of DNF Logic Testing
4
Minimal DNF Summary
Terms separated by OR, literals by AND ab + a!c vs. a(b + !c)
• Make each term true and other terms false ab + ac vs. ab + abc
• Impossible to remove a literal without changing the predicate semanticsab vs. abc + ab!c
Green – in minimal DNF Red – not in minimal DNF
5
Nine Minimal DNF Logic Faults
Original: ab + bc
• Expression Negation Fault (ENF) !(ab + bc)• Operator Reference Fault (ORF) abbc• Operator Reference Fault (ORF) a + b + bc• Term Negation Fault (TNF) !(ab) + bc• Term Omission Fault (TOF) bc• Literal Negation Fault (LNF) !ab + bc• Literal Insertion Fault (LIF) abc + bc• Literal Reference Fault (LRF) ac + bc • Literal Omission Fault (LOF) a + bc
Fault Detection Hierarchy
Expression Negation Fault
ENF
Literal Insertion FaultLIF
Term Omission FaultTOF
Literal Reference Fault
LRF
Literal Negation FaultLNF
Operator Reference Fault
ORF+
Literal Omission FaultLOF
Term Negation FaultTNF
Operator Reference Fault
ORF*
6
Key Fact:Detection Relations Depend on Whether Fault is Feasible
7
Coverage Criteria Preamble : Unique True Points and Near False Points
• UTP: An assignment of values such that only one term evaluates to true.
ab + bc: TTF is a UTP for term ab
• NFP: An assignment of values such that the predicate evaluates to false but when a literal is omitted, it evaluates to true.
ab + bc: TFF is an NFP for b in term ab
Closely Related to MCDC Notion
• For each implicant find unique true points (UTPs) so that– Literals not in term take on values T and F
• Consider the DNF predicate: – f = ab + cd
• For term ab– Choose TTFT, TTTF
• For term cd– Choose FTTT, TFTT
• MUTP test set– {TTFT, TTTF, FTTT, TFTT}
MUTP Criterion (Chen, Lau, Yu)
01
00
10110100 ab cd
t
t
tt11
10
t
tt
CUTPNFP Criterion (Chen, Lau, Yu)• Consider f = ab + cd• For term ab
– For a, choose UTP, NFP pair• TTFF, FTFF
– For b, choose UTP, NFP pair• TTFT, TFFT
• For term cd– For c, choose UTP, NFP pair
• FFTT, FFFT– For d, choose UTP, NFP pair
• FFTT, FFTF• Possible CUTPNFP test set
– {TTFF, TTFT, FFTT //UTPs
FTFF, TFFT, FFFT, FFTF} //NFPs
01
00
10110100 ab cd
t
t
tt11
10
t
tt
• Find NFP tests for each literal such that all literals not in the term attain F and T
• Consider the DNF predicate: – f = ab + cd
• For implicant ab– Choose FTFT, FTTF for a– Choose TFFT, TFTF for b
• For implicant cd– Choose FTFT, TFFT for c– Choose FTTF, TFTF for d
• MNFP test set– {TFTF, TFFT, FTTF, TFTF}
• Example is small, but generally MNFP is large
MNFP Criterion (Chen, Lau, Yu)
01
00
10110100 ab cd
t
t
tt11
10
t
tt
11
MUMCUT/Minimal-MUMCUT Criteria• MUMCUT is simply MUTP + CUTPNFP + MNFP• Minimal-MUMCUT omits tests via low level criterion feasibility analysis
– Adds CUTPNFP and MNFP tests only when necessary• Minimal-MUMCUT guarantees detecting LIF, LRF, LOF
– And thus all 9 faults in the hierarchy, even if some faults infeasible
Key Point: Minimal != MinimumLocal vs. Global Perspective
CUTPNFPfeasible?
MNFPTest Set =
MUTP + MNFP
For Each
Literal In Term
Test Set =MUTP + CUTPNFP
MUTP feasible?
Test Set =MUTP + NFP
For Each Term
12
MUTP Optimization
• Minimize test set size subject to detecting LRFs with UTPs– LIFs Guaranteed Be Detected When LRFs are Feasible
• Each UTP is a 0-1 variable (integer programming)• Each constraint represents the UTPs that detect a LRF• Can reduce number of variables and constraints by focusing on a fault
called the TRF-LIF instead of the LRF (see paper for details)
ab + cd
Minimize x1100 + x1101 + x1110 + x0011 + x0111 + x1011
Subject to
x1100 + x1101 >= 1 (cb + cd)
x1110 >= 1 (!cb + cd)
x1100 + x1110 >= 1 (db + cd)
x1101 >= 1 (!db + cd)
…….
13
CUTPNFP Optimization• Minimize test set size subject to finding UTP – NFP corresponding pairs• Each point is a 0-1 variable• Constraints represent an implication that the UTP and NFP be corresponding• Additional constraint to reflect that an UTP must be selected
ab + cd
Minimize x1100 + x0100 + x1000 + x1101 + …
Subject To• x1100 <= M1; -- if 1100 is selected as an UTP then M1 must = 1• x0100 >= M1; -- if M1 = 1, 0100 must be chosen as a NFP • x1000 >= M1; -- if M1 = 1, 1000 must be chosen as a NFP • x1101 <= M2; -- if 1101 is selected as an UTP then M2 must = 1 • x0101 >= M2; -- if M2 = 1, 0101 must be chosen as a NFP• x1001 >= M2; -- if M2 = 1,1001 must be chosen as a NFP • x1110 <= M3; -- if 1110 is selected as an UTP then M3 must = 1 • x0110 >= M3; -- if M3 = 1, 0110 must be chosen as a NFP • x1010 >= M3; -- if M3 = 1, 1010 must be chosen as a NFP • x1100 + x1101 + x1110 >= 1; -- an UTP must be selected
14
MNFP Optimization
• Minimize test set size subject to detecting LRFs with NFPs• Each NFP is a 0-1 variable• Each constraint represents the NFPs that detect a LRF
ab + cd
Minimize x0100 + x0101 + x0110 + x1000 + x1001 + x1010 ...
Subject to
x0110 >= 1 (cb + cd)
x0100 + x0101 >= 1 (!cb + cd)
x1001 >= 1 (db + cd)
x1000 + x1010 >= 1 (!db + cd)
…….
15
Minimal-MUMCUT Optimization
• Applies optimization techniques for MUTP, CUTPNFP, and MNFP above based on criterion feasibility
• When MUTP is feasible, can select any NFP• When MUTP is not feasible, but CUTPNFP is feasible, apply
CUTPNFP optimization• Otherwise, apply MNFP optimization
ab + cd
Minimize x0100 + x0101 + x0110 + x1000 + x1001 + x1010 + …
Subject to
x0100 + x0101 + x0110 >= 1 (any NFP for a works as MUTP is feasible)
x1000 + x1001 + x1010 >= 1 (any NFP for b works as MUTP is feasible)
…….
16
Case Study• Analyzed 19 Boolean predicates in an avionics software
system (Weyuker, Chen, Lau, and Yu)
• Number of unique literals range: 5 to 13
• Built a tool in Java to automatically:- generate MUTP, CUTPNFP, MNFP, Minimal-MUMCUT
tests based on heuristics- generate an optimization model that when solved by
CPLEX yields minimized test sets
• Examined test set size and computation time difference between heuristics and optimization
17
Average Test Set Size Results
Criterion Minimized Test Set Size
Heuristic Test Set Size
Percentage
MUTP 15.00 15.00 100.00%
CUTPNFP 41.16 44.79 91.90%
MNFP 37.89 46.16 82.08%
Minimal-MUMCUT
40.37 43.95 91.85%
18
Average Computation Time Results (sec)
Criterion Heuristics Optimization(Model Generation
and Solver)
MUTP 0.07 0.23
CUTPNFP 0.11 1.33
MNFP 0.22 5.95
Minimal-MUMCUT
5.15 6.09
19
Average Optimization Model Size
Criterion Number of Variables
Number of Constraints
MUTP 15 15
CUTPNFP 561 336
MNFP 414 245
Minimal-MUMCUT
392 106
20
Interpretation of Results
• For a small difference in computation time, optimization produces smaller test sets
• Running tests is time-consuming and expensive, so reducing test set size by one can be significant
• Choose optimization over heuristics
21
Fault Detection Maximization• Testers have limitations on the number of tests they can run and thus it
may be impossible to satisfy a criterion. In this case, it would help to know what tests maximize fault detection subject to a given test set size.
• Each point (test) is a 0-1 variable - x1100 represents the point (test) where a is T, b is T, c is F, d is F - x1100 will be 0 if it is not in the test set and 1 if it is in the test set
• Each fault is a 0-1 variable - LIF_1_c represents a fault where literal c is inserted into term 1 - LIF_1_c will be 0 if it is not detected and 1 if it is detected - objective function is to maximize the sum of all the fault variables - a fault variable exists for each fault based on the 9 DNF fault types
22
Fault Detection MaximizationConsider ab + cd
Maximize …. + LIF_1_c + ...
Subject to• x1101 + x1100 >= M_LIF_1_c; (1)• LIF_1_c <= M_LIF_1_c; (2)……
LIF_1_c represents a fault where c is inserted into term 1 (ab).
If neither 1101 nor 1100 is selected, the x1101 and x1100 variables will be 0. For constraint (1) to hold, the variable M_LIF_1_c must then be 0.Thus, variable LIF_1_c must be 0 as well and the fault goes undetected.
If 1101 or 1100 is selected, either x1101 and x1100 will be 1. This allows the variable M_LIF_1_c to be 1, which in turn allows LIF_1_c to be 1 and thus the fault is detected.
23
Case Study• Used same predicates as in the first study
• Built a tool in Java to automatically:- generate an optimization model that when
solved yields a test set that maximizes fault detection subject to a given test set size
- generate an optimization model that when solved yields a test set that minimizes fault detection subject to a given test set size (by flipping constraints in the model)
• Used the tool to compare test set size between a best, random, and worst case scenario
24
Case Study Results
Amongst all 19 predicates:• As many as 288 times more faults are detected in the
best case than the worst case for a single test.• As many as 7 times more faults are detected in the best
case than the random case for a single test.
Implication for regression testing
Scenario Fault Detection Percentage for 1 test
Worst 0.35%
Random 6.13%
Best 15.97%
25
Case Study Results
0102030405060708090
100
Percentage
of Faults Detected
1 5 10 15 MIN
Test Set Size
Random
Best
26
Fraser and Gargantini Study• Found fault detecting tests for the same 19 predicates by formalizing
the problem as a logic satisfiability problem which can be solved by a SAT algorithm (not optimization)
• 3 approaches1) Minimal-MUMCUT heuristic every test needed for fault detection2) Satisfiability Algorithm improves on heuristic3) Optimization guaranteed smallest test set size
Minimal-MUMCUT
Satisfiability Algorithm
Optimization
Avg Test Set Size 43.95 42.37 40.37
27
Conclusion
• Applied optimization techniques in the Minimal DNF logic testing domain to:
1) Minimize test set size subject to guaranteeing fault detection (by satisfying a given criterion)
2) Maximize fault detection subject to a certain test set size
• Case study showed for a small sacrifice in computation time, optimization produced smaller test sets
• Case study showed large difference in fault detection between a best, random, and worst case scenario
• Applications in testing of programs with large predicates
28
Thoughts On Where To Go Next• Reformulating Model To Handle Infeasible Tests
– Crucial for Handling Predicates Embedded In Code– Requires Removal Of Some Constraints
• Some Faults Become May Become Equivalent• Hence Constraint To Trigger Fault is Infeasible
– Also Requires Adding Additional Constraints• Fault Hierarchy Only Applies For Feasible Faults
• Applying Optimization To Other Coverage Criteria– Eg: Branch Coverage
• Building Optimal “Test Databases”– Given a Query With A Particular “Where” Clause– What Minimal Database Guarantees Detecting All 9 Faults in
Query?• Tool Available at Ammann/Offutt Book Web Site
– http://cs.gmu.edu/~offutt/softwaretest