software product line testing part iv : a framework for variability coverage
DESCRIPTION
Software Product Line Testing Part IV : A Framework for Variability Coverage. Myra CohenMatthew Dwyer Laboratory for Empirically-based Software Quality Research Department of Computer Science University of Nebraska - Lincoln. Work supported by NSF CCF through awards 0429149 and 0444167, by - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/1.jpg)
e2 ESQuaReD
Software Product Line TestingPart IV : A Framework for
Variability CoverageMyra Cohen Matthew Dwyer
Laboratory for Empirically-based Software Quality Research
Department of Computer Science
University of Nebraska - Lincoln
Work supported by NSF CCF through awards 0429149 and 0444167, by the U.S. Army Research Office through award DAAD190110564 and by an NSF EPSCoR FIRST award.
![Page 2: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/2.jpg)
2
OutlineSoftware Product Lines : What and Why?
Modeling Variability in Software Product Lines
Validating Product Lines
A Framework for Variability Coverage
Toward Product Line Driven Test Processes
![Page 3: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/3.jpg)
3
Outline
A Framework for Variability Coverage
1. Adapting Covering Arrays to Real Systems2. Building Covering Arrays
1. Overview 2. One-row-at-a-time Greedy Algorithms3. Meta-heuristic Search
![Page 4: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/4.jpg)
4
CA(9;2,4,3) ( also an OA(2,4,3))
Config Browser OS Connection Printer
1 Netscape Windows XP LAN Local
2 Netscape Linux ISDN Networked
3 Netscape OS X PPP Screen
4 IE Windows XP ISDN Screen
5 IE OS X LAN Networked
6 IE Linux PPP Local
7 Mozilla Windows PPP Networked
8 Mozilla Linux LAN Screen
9 Mozilla OS X ISDN Local
A set of product line instances that covers all pair-wise interactions.
![Page 5: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/5.jpg)
5
Interaction Strength
• We can quantify the “coverage” for a particular interaction strength.
• Example:– 4 factors– Each has 3 values– Quantify 2-way coverageAny single test case can cover at most or 6 possible pairs.
The system has or 54 pair-wise interactions.
The addition of one new test can contribute at most 6/54 or 11.1% pair wise coverage.
⎟⎟⎠
⎞⎜⎜⎝
⎛2
4
232
4⎟⎟⎠
⎞⎜⎜⎝
⎛
![Page 6: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/6.jpg)
6
Adaptations for Real Systems
• So far we have seen a covering array with v symbols. This means each factor has the same number of values.
BUT:• This is usually not the case in a real system.
•Model : Civic, Accord•Package : Sedan, Coupe, Hybrid, GX, Si•Transmission : manual, auto, cvt•Power : gas, hybrid, natural gas•Doors : 2, 4•Cylinders : 4, 6•Nav system : Y/N
![Page 7: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/7.jpg)
7
Mixed Level Covering ArraysMCA(N;t,k,(v1,v2,…,vk)) Is an N x k array on v symbols where:
v =
And:– For each column i where (i i k)– The rows of each N x t sub-array cover all t-tuples or values
from the t columns at least times.
∑=
k
i iv1
Shorthand Notation:
MCA(N;t,(w1k1w2
k2 …wsks))
Example: MCA(12;2,4,(4, 3,3,2)) MCA(12;2,(41 3221))
![Page 8: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/8.jpg)
8
MCA(12;2,413221)
Netscape Windows XP LAN Local
Mozilla OS X ISDN Networked
Safari Linux PPP Networked
Mozilla Linux LAN Local
Netscape OS X PPP Local
IE Windows XP ISDN Networked
IE OS x LAN Networked
Safari Windows XP ISDN Networked
Netscape Linux ISDN Local
Mozilla Windows XP PPP Local
Safari OS X LAN Local
IE Linux PPP Networked
![Page 9: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/9.jpg)
9
MCA(12;2,413221)
Netscape Windows XP LAN Local
Mozilla OS X ISDN Networked
Safari Linux PPP Networked
Mozilla Linux LAN Local
Netscape OS X PPP Local
IE Windows XP ISDN Networked
IE OS x LAN Networked
Safari Windows XP ISDN Networked
Netscape Linux ISDN Local
Mozilla Windows XP PPP Local
Safari OS X LAN Local
IE Linux PPP Networked
![Page 10: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/10.jpg)
10
Limitation
• Mixed level covering arrays are closer to real systems; they can be used for an arbitrary software system.
• But they view a system flatly. They force a (perhaps arbitrary) restriction on the importance of various parts of the system.
![Page 11: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/11.jpg)
11
Scenarios1. When testing a software system certain
components may be closely interrelated. This may be determined by static analysis.
2. Operational profiles give us information that certain areas of the system are used more often than others.
3. In modifying a system only certain regions are changed therefore we want to test more strongly in this area.
4. Failures in certain parts of a system are more costlier than in others.
![Page 12: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/12.jpg)
12
Possible Models
t=2t=3
t=4
![Page 13: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/13.jpg)
13
Some Possible Models
t=2t=3
t=2t=3
t=4
t=3
t=4
![Page 14: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/14.jpg)
14
An Initial Model
strength 2 array
A (Browser)
NetscapeIEMozilla
C(Connection)LANPPP
B (OS)
Windows XPOS XLinux
D (Printer)LocalNetworked
Strength 3 array
![Page 15: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/15.jpg)
15
Variable Strength Covering Arrays
MCA(9;2,3222)
B
D
A
C
0,1,2
3,4,5
6,7 8,9
A B C D
0 4 7 8
1 3 7 9
2 3 6 8
0 5 6 9
1 4 6 8
2 4 7 9
2 5 7 9
1 5 7 8
0 3 6 9
t=2t=3
A 3-way array would have 18 rows
![Page 16: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/16.jpg)
16
Variable Strength Covering Arrays
MCA(9;2,3222)
Additional rows to guarantee subset of 3-way coverage
B
D
A
C
0,1,2
3,4,5
6,7 8,9
A B C D
0 4 7 8
1 3 7 9
2 3 6 8
0 5 6 9
1 4 6 8
2 4 7 9
2 5 7 9
1 5 7 8
0 3 6 9
0* 3 7 8
2* 4 6 9
1* 5 6 8
t=2t=3
![Page 17: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/17.jpg)
17
Variable Strength Covering Array
• A VCA(N;t,k,(v1,v2,…vk), C) is a t-way mixed level covering array on v symbols with a vector, C, of covering arrays each with strength > t and defined on a subset of the k columns of the VCA.
![Page 18: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/18.jpg)
18
V
VP
IntrusionDetection
VP
DoorLocks
CameraSurveillance
MotionSensors
CulletDetection Basic Advanced Keypad
FingerprintScanner
SecurityPackage
VP
V V V V V V
requires_v_v
requires_v_v
Adding Constraints
![Page 19: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/19.jpg)
19
Intrusion Detection A Intrusion Detection B
Security Package
Door Locks
Camera Surveillance None Advanced Fingerprint Scanner
Motion Sensors Cullet Detection Advanced Fingerprint Scanner
Camera Surveillance Cullet Detection Advanced Keypad
Motion Sensors None Basic Keypad
Camera Surveillance Cullet Detection Basic Fingerprint Scanner
Constrained Set of Product Instances “Basic requires Motion Sensors and Keypad”
“Basic excludes Cullet Detection”
Adding Constraints
![Page 20: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/20.jpg)
20
Intrusion Detection A Intrusion Detection B
Security Package
Door Locks
Camera Surveillance None Advanced Fingerprint Scanner
Motion Sensors Cullet Detection Advanced Fingerprint Scanner
Camera Surveillance Cullet Detection Advanced Keypad
Motion Sensors None Basic Keypad
Camera Surveillance Cullet Detection Basic Fingerprint Scanner
Constrained Set of Product Instances “Basic requires Motion Sensors and Keypad”
“Basic excludes Cullet Detection”
Adding Constraints
![Page 21: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/21.jpg)
21
Intrusion Detection A Intrusion Detection B
Security Package
Door Locks
Camera Surveillance None Advanced Fingerprint Scanner
Motion Sensors Cullet Detection Advanced Fingerprint Scanner
Camera Surveillance Cullet Detection Advanced Keypad
Motion Sensors None Basic Keypad
Camera Surveillance Cullet Detection Basic Fingerprint Scanner
Constrained Set of Product Instances “Camera Surveillance requires Fingerprint Scanner”
Another set of Constraints
![Page 22: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/22.jpg)
22
Intrusion Detection A Intrusion Detection B
Security Package
Door Locks
Camera Surveillance None Advanced Fingerprint Scanner
Motion Sensors Cullet Detection Advanced Fingerprint Scanner
Camera Surveillance Cullet Detection Advanced Keypad
Motion Sensors None Basic Keypad
Camera Surveillance Cullet Detection Basic Fingerprint Scanner
Motion Sensors Cullet Detection Advanced Keypad
Constrained Set of Product Instances “Camera Surveillance requires Fingerprint Scanner”
Another set of Constraints
![Page 23: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/23.jpg)
23
Constraints in our Original Example
“Mozilla does not run on Linux”“Linux does not support print to the screen”
Test Case Browser OS Connection Printer
1 Netscape Windows XP LAN Local
2 Netscape Linux ISDN Networked
3 Netscape OS X PPP Screen
4 IE Windows XP ISDN Screen
5 IE OS X LAN Networked
6 IE Linux PPP Local
7 Mozilla Windows XP PPP Networked
8 Mozilla Linux LAN Screen
9 Mozilla OS X ISDN Local
![Page 24: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/24.jpg)
24
Constraints in our Original Example
“Mozilla does not run on Linux”“Linux does not support print to the screen”
Test Case Browser OS Connection Printer
1 Netscape Windows XP LAN Local
2 Netscape Linux ISDN Networked
3 Netscape OS X PPP Screen
4 IE Windows XP ISDN Screen
5 IE OS X LAN Networked
6 IE Linux PPP Local
7 Mozilla Windows XP PPP Networked
8 Mozilla Linux LAN Screen
8 Mozilla OS X ISDN Local
9 IE Linux LAN Local
10 Netscape Windows XP LAN Screen
![Page 25: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/25.jpg)
25
Other Practical Issues
• Seeded or default instances.
• Aggregate factors.
• Cost of testing specific instances.– May consider order of validating instances and set-
up costs.
![Page 26: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/26.jpg)
26
Outline : Interaction Testing in Practice
A Framework for Variability Coverage
1. Adapting Covering Arrays to Real Systems2. Building Covering Arrays
1. Overview 2. One-row-at-a-time Greedy Algorithms3. Meta-heuristic Search
![Page 27: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/27.jpg)
27
Combinatorial Results
• How do we know the covering array number (CAN)?
• How can we find the a subset of product instances that satisfies the properties of a covering array?
![Page 28: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/28.jpg)
28
Results on Covering Arrays
There are two types of results:1. Probabilistic:
• We can prove a bound exists but can’t necessarily create the subset. Only useful for finding the CAN.
![Page 29: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/29.jpg)
29
Results on Covering Arrays
There are two types of results:2. Constructive:
• prove a new bound by giving a direct or algebraic construction.
–Often recursive in nature. –Requires extensive mathematical
knowledge.–Only works for certain values of t, k, v.
![Page 30: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/30.jpg)
30
Constructive Results
• Although direct or algebraic constructions often give us the smallest subset for a covering array, they are not general.
• There are less constructions for mixed level covering arrays and none yet for variable strength arrays.
![Page 31: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/31.jpg)
31
Computational Search
• These are constructive techniques. Pros:– They are general. – They extend easily to mixed level and can be
adapted for variable strength, and seeds. Cons:– May not always give us the optimal CAN size.– May take a long computational time to find a
CA.
![Page 32: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/32.jpg)
32
Computational Search
• The problem of finding whether a minimal covering array exists for a given size is a difficult problem.
• Variants of this problem have been formulated and shown to be NP Hard problems.
• We cannot exhaustively search for a solution!
![Page 33: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/33.jpg)
33
Algorithmic TechniquesGreedy algorithms: • AETG - Automatic Efficient Test Case Generator• TCG - Test Case Generator • DDA -Deterministic Density Algorithm• IPO - In Parameter Order Heuristic/Meta-heuristic Search:• Hill Climbing• Simulated Annealing• Genetic Algorithms• Tabu Search
![Page 34: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/34.jpg)
34
Some Available Tools• AETG - Telcordia, Inc.: Commercial product
http://aetgweb.argreenhouse.com/
• WHITCH - Alan Hartman: Eclipse plugin. It includes a tool called CTS (uses algebraic constructions) and a tool called tofu. You can plug in your own algorithms as well.
http://alphaworks.ibm.com/tech/whitch
• TestCover - George Sherwood: Commercial tool: uses algebraic constructions. Has a student use license available.
http://www.testcover.com/
• TConfig - Alan Williams: Uses algebraic constructions. Also includes an implementation of IPO.
http://www.site.uottawa.ca/~awilliam/
![Page 35: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/35.jpg)
35
Outline : Interaction Testing in Practice
A Framework for Variability Coverage
1. Adapting Covering Arrays to Real Systems2. Building Covering Arrays
1. Overview 2. One-row-at-a-time Greedy Algorithms3. Meta-heuristic Search
![Page 36: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/36.jpg)
36
Greedy Algorithms
• One class of greedy algorithms add one-row at a time until the covering array is complete.
• The algorithms included in this brief overview include: AETG, TCG, DDA
![Page 37: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/37.jpg)
37
One Row at a Time Greedy Algorithms
. . .
. . .
. . .
M Candidates
y z . . . x1…………………………….k
use factor ordering to select values
……
N
1... add rows until all
t-sets covered
select best instance to add
![Page 38: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/38.jpg)
38
AETG
• The commercial AETG tool is patented.• It uses some algebraic constructions,
handles constraints and includes some post-processing steps.
• The original algorithm is presented next, but it does not include these improvements.
![Page 39: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/39.jpg)
39
AETG Summary
• For each test case create M (50) new test cases.
• For each of the M test cases– Permute the order of the factors so that the first column has a
symbol with the largest number of uncovered pairs. We fix this symbol and column. Randomly permute the rest.
– Fill in the values for factors in the permutation order• For each factor choose the symbol that creates the most new
pairs with the other columns already filled.
• Select the product instance (from M) which covers the most new pairs.
![Page 40: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/40.jpg)
40
AETG
Pairs Covered:3,6 0,53,7 0,83,10 0,116,7 5,86,10 5,117,10 8,11
Factor Ordering: 3,2,4,1
![Page 41: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/41.jpg)
41
AETG
Pairs Covered:3,6 0,53,7 0,83,10 0,116,7 5,86,10 5,117,10 8,11
Factor Ordering: 3,2,4,1
SELECT Value 10
![Page 42: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/42.jpg)
42
Other Members of the Framework
• TCG - Test Case Generator
(Tung and Aldiwan)
• DDA - Deterministic Density Algorithm (Colbourn, Cohen, Bryce)
![Page 43: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/43.jpg)
43
TCG
• Sorts factors in decreasing order of number of values.
• Always fills in the values in this order.
• When there is a tie between two values the one that has been used least often is selected.
![Page 44: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/44.jpg)
44
TCG
![Page 45: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/45.jpg)
45
TCG
SELECT value 5
![Page 46: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/46.jpg)
46
Outline : Interaction Testing in Practice
A Framework for Variability Coverage
1. Adapting Covering Arrays to Real Systems2. Building Covering Arrays
1. Overview 2. One-row-at-a-time Greedy Algorithms3. Meta-heuristic Search
![Page 47: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/47.jpg)
47
Heuristic Search
• Heuristic techniques: methods that seek a good (close to optimal) solution using a reasonable amount of computational cost. These techniques cannot guarantee optimality and may not guarantee feasibility.
![Page 48: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/48.jpg)
48
Heuristic Search
Combinatorial Optimization algorithms1) Define a Set of Feasible solution2) Each solution assigned a cost3) Perform a series of transitions to new
solutions chosen at random4) If the new solution has the same or better cost
– commit the change.5) Otherwise don’t commit.6) We continue until we have an optimal cost or
we are frozen
![Page 49: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/49.jpg)
49
Hill Climbing
∑ -Set of feasible solutions
cost(S) : number of uncovered t-sets in S
When cost=0 we have a covering array
S1 S2
S3 ….. Si
•Start with a random feasible solution
•Randomly choose/change one symbol to form a new feasible solution
![Page 50: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/50.jpg)
50
Random instance
Random factor
Cost
# uncovered t-sets Change value
Example
1…………………………..k
1
.
.
.
.
N
![Page 51: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/51.jpg)
51
Example of Hill ClimbRows
1 0 1 1
2 1 0 1
3 0 1 0
4 1 1 0
5 0 1 1
6 1 0 0
7 0 0 0
8 0 0 0
CA(N;3,3,2)
t-sets000 - 2 001 - 0010 - 1011 - 2100 - 1101 - 1110 - 1 111 - 0
Cost = 2 Change?
![Page 52: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/52.jpg)
52
Example of Hill ClimbRows
1 0 1 1
2 1 0 1
3 0 1 0
4 1 1 0
5 1 1 1
6 1 0 0
7 0 0 0
8 0 0 0
CA(N;3,3,2)
t-sets000 - 2 001 - 0010 - 1011 - 2 1100 - 1101 - 1110 - 1 111 - 0 1
Cost = 1 Improves the Cost - keep this change
![Page 53: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/53.jpg)
53
Example of Hill ClimbRows
1 0 1 1
2 1 0 1
3 0 1 0
4 1 1 0
5 0 1 1
6 1 0 0
7 0 0 0
8 0 0 0
CA(N;3,3,2)
t-sets000 - 2 001 - 0010 - 1011 - 2100 - 1101 - 1110 - 1 111 - 0
Cost = 2Change?
![Page 54: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/54.jpg)
54
Example of Hill ClimbRows
1 0 1 1
2 1 0 1
3 0 1 0
4 1 1 0
5 0 1 1
6 1 1 0
7 0 0 0
8 0 0 0
CA(N;3,3,2)
t-sets000 - 2 001 - 0010 - 1011 - 2100 - 1 0101 - 1110 - 1 2 111 - 0
Cost = 3Makes the solution worse! Do not keep
![Page 55: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/55.jpg)
55
Lower bound L
Binary search for “best” solution
Upper bound U
(L+U)/2
Finding “N”
![Page 56: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/56.jpg)
56
Problems with Hill Climbing
• May get stuck in local optimums.
![Page 57: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/57.jpg)
57
Meta-heuristic Algorithms
• Provide mechanisms to escape local optimums. Sometimes we make accept a worse choice in the hope we will find a better route to a good solution.
• We want to control the probability of making a bad choice.
![Page 58: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/58.jpg)
58
Some Examples
• Genetic Algorithms
• Simulated Annealing
• Tabu Search
• Ant Colony Algorithm
![Page 59: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/59.jpg)
59
Simulated Annealing
• Based on the physical annealing process that is used to cool metal.
• Idea is to start at a high temperature. At each temperature the molecules stabilize and then the metal is cooled to the next temperature.
![Page 60: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/60.jpg)
60
S
t(S)
N(S)
Select t(S) such that
cost(t(S)) ≤ cost(S) or
cost(t(S)) > cost(S) subject to controlled probability
Search space
Simulated Annealing
![Page 61: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/61.jpg)
61
Metropolis condition:
Accept bad move with probability e-/T
Design decisions (cooling schedule):
(1) Value of T0
(2) Generating Tk from Tk-1
(3) Length of Markov chain Lk
(4) Stopping condition
Simulated Annealing
![Page 62: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/62.jpg)
62
Other Meta-heuristic Search
• Genetic algorithms: population based search. (J. Stardom 2000)
• Tabu Search: steepest descent. Maintain a tabu list to prevent getting stuck in a cycle. (K. Nurmela 2004)
![Page 63: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/63.jpg)
63
References
D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, The AETG system: an approach to testing based on combinatorial design, IEEE Transactions on Software Engineering, vol. 23, no.7, pp. 437--444, 1997.
M. B. Cohen, C.J. Colbourn, J.Collofello, P.B. Gibbons, and W. B. Mugridge,Variable strength interaction testing of components in 'Proc. of 27th Intl. Computer Software and Applications Conference (COMPSAC), November 2003, pp. 413--418.
M.B. Cohen, C.J. Colbourn, P.B. Gibbons, and W.B. Mugridge, Constructing test suites for interaction testing, in Proc. of the Intl. Conf. On Software Engineering (ICSE), May 2003, pp. 38--48.
M. B. Cohen, C. J. Colbourn, and A. C. H. Ling, Constructing strength three covering arrays with augmented annealing, Discrete Mathematics, to appear.
M..B. Cohen, Designing test suites for software interaction testing, PhD Dissertation, University of Auckland, Department of Computer Science, 2004.
![Page 64: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/64.jpg)
64
References
C.J. Colbourn, M. B. Cohen, and R..C. Turban, A deterministic density algorithm for pairwise interaction coverage, in IASTED Proc. of the Intl. Conference on Software Engineering, February 2004, pp. 345--352.
A.Hartman and L.Raskin, Problems and algorithms for covering arrays, Discrete Math, vol. 284, pp. 149 -- 156, 2004.
K.J. Nurmela, Upper bounds for covering arrays by tabu search, Discrete Applied Mathematics, vol. 138, no. 1-2, pp. 143--152, 2004.
J. Stardom, Metaheuristics and the search for covering and packing arrays, Master's thesis, Simon Fraser University, 2001
K..C. Tai and Y. Lei, A test generation strategy for pairwise testing, IEEE Transactions on Software Engineering, vol.28, no.1, pp. 109--111, 2002.
![Page 65: Software Product Line Testing Part IV : A Framework for Variability Coverage](https://reader035.vdocument.in/reader035/viewer/2022070416/568150a9550346895dbebbd0/html5/thumbnails/65.jpg)
65
References
Y.-W. Tung and W. S. Aldiwan, Automating test case generation for the new generation mission software system,'' in Proc. IEEE Aerospace Conf., 2000, pp. 431--437.
A. W. Williams, Determination of test configurations for pair-wise interaction coverage, in Thirteenth Intl. Conf. Testing Communication Systems, 2000, pp. 57--74.