search based testing - for webtarot2010.ist.tugraz.at/slides/mcminn.pdf · overview how and why...
TRANSCRIPT
![Page 1: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/1.jpg)
Search-Based TestingPhil McMinn
University of Sheffield, UK
![Page 2: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/2.jpg)
Overview
How and why Search-Based Testing Works
Examples: Temporal, Functional, Structural
Search-Based Test Data Generation
Empirical and Theoretical Studies
Testability Transformation
Input Domain Reduction
![Page 3: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/3.jpg)
Acknowledgements
The material in some of these slides has kindly been provided by:
Mark Harman (KCL)
Joachim Wegener (Berner & Matner)
![Page 4: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/4.jpg)
the hard part...
manual design of test cases / scenarios
laborious,time-consuming
tedious
difficult!(where are the faults ???)
Conventional Testing
![Page 5: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/5.jpg)
Search-Based Testing is an automated search of a
potentially large input space
the search is guided by a problem-specific ‘fitness function’
the fitness function guides the search to the test goal
![Page 6: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/6.jpg)
Random Test Data Generation
Input
![Page 7: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/7.jpg)
Fitness-guided search
Input
Fitn
ess
![Page 8: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/8.jpg)
Fitness-guided search
Input
Fitn
ess
![Page 9: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/9.jpg)
Fitness Function
The fitness function scores different inputs to the system according to the test goal
which ones are ‘good’ (that we should develop/evolve further)
which ones are useless (that we can forget about)
![Page 10: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/10.jpg)
Publications since 1976
source: SEBASE publications repository http://www.sebase.org
![Page 11: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/11.jpg)
Fitness Functions
Often easy
We often define metrics
Need not be complex
![Page 12: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/12.jpg)
Conventional testingmanual design of test cases / scenarios
laborious,time-consuming
tediousdifficult!
(where are the faults ???)
Search-Based Testing:automatic - may sometimes be time consuming, but it is not a human’s time being consumed
Search-Based Testing:a good fitness function will lead the search to
the faults
![Page 13: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/13.jpg)
Generating vs CheckingConventional Software Testing Research
Write a method to construct test cases
Search-Based Testing
Write a method to determine how good a test case is
![Page 14: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/14.jpg)
Generating vs CheckingConventional Software Testing Research
Write a method to construct test cases
Search-Based Testing
Write a fitness functionto determine how good a test case is
![Page 15: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/15.jpg)
White box testing
fitness function analysesthe outcome of
decision statements and the values of variables in
predicates
![Page 16: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/16.jpg)
White-box + testing for assertion violations
safety condition (desired state):
speed < 150mph
fitness function:f =150 - speed
fitness minimised If f is zero or less a fault is found
![Page 17: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/17.jpg)
Search Techniques
![Page 18: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/18.jpg)
Hill Climbing
Input
Fitn
ess
![Page 19: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/19.jpg)
Hill Climbing
Input
Fitn
ess
No better solution in neighbourhood Stuck at a local optima
![Page 20: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/20.jpg)
Hill Climbing - Restarts
Input
Fitn
ess
![Page 21: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/21.jpg)
Hill Climbing - Restarts
Input
Fitn
ess
![Page 22: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/22.jpg)
Simulated Annealing
Input
Fitn
ess
Worse solutions temporarily
accepted
![Page 23: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/23.jpg)
Evolutionary Algorithm
Input
Fitn
ess
![Page 24: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/24.jpg)
Evolutionary Algorithm
Input
Fitn
ess
![Page 25: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/25.jpg)
Evolutionary Testing
Mutation
Recombination
Selection
Insertion
Fitness Evaluation
End?
Test cases
Monitoring
Execution
![Page 26: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/26.jpg)
Which search method?Depends on characteristics of the search landscape
![Page 27: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/27.jpg)
Which search method?Some landscapes are
hard for some searches but easy for
others
more on this later...
...and vice versa...
![Page 28: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/28.jpg)
Representation
Fitness Function
Neighbourhood
Ingredients for an optimising search
algorithm
![Page 29: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/29.jpg)
Representation
Fitness Function
Neighbourhood
Ingredients for Search-Based Testing
![Page 30: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/30.jpg)
Representation A method of encoding all possible inputs
Usually straightforward
Inputs are already in data structuresFitness Function
Neighbourhood
Ingredients for Search-Based Testing
![Page 31: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/31.jpg)
Part of our understanding of the problem
We need to know our near neighbours
Representation
Fitness Function
Neighbourhood
Ingredients for Search-Based Testing
![Page 32: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/32.jpg)
Transformation of the test goal to a numerical function
Numerical values indicate how ‘good’ an input is
Ingredients for Search-Based Testing
Representation
Fitness Function
Neighbourhood
![Page 33: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/33.jpg)
More search algorithms
Tabu Search
Estimation of Distribution Algorithms
Particle Swarm Optimisation
Ant Colony Optimisation
Genetic Programming
![Page 34: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/34.jpg)
Important Publications
Phil McMinn: Search-based software test data generation: a survey. Software Testing, Verification and Reliability 14(2): 105-156, 2004
Wasif Afzal, Richard Torkar and Robert Feldt: A Systematic Review of Search-based Testing for Non-Functional System Properties. Information and Software Technology, 51(6):957-976, 2009
Shaukat Ali, Lionel Briand, Hadi Hemmati and Rajwinder Panesar-Walawege: A Systematic Review of the Application and Empirical Investigation of Search-Based Test-Case Generation. IEEE Transactions on Software Engineering, To appear, 2010
M. Harman and B. Jones: Search-based software engineering. Information and Software Technology, 43(14):833–839, 2001.
M. Harman: The Current State and Future of Search Based Software Engineering, In Proceedings of the 29th International Conference on Software Engineering (ICSE 2007), 20-26 May, Minneapolis, USA (2007)
D. Whitley: An overview of evolutionary algorithms: Practical issues and common pitfalls. Information and Software Technology, 43(14):817–831, 2001.
O Bühler and J Wegener: Evolutionary functional testing, Computers & Operations Research, 2008 - Elsevier
J. Wegener and M. Grochtmann. Verifying timing constraints of real-time systems by means of evolutionary testing. Real-Time Systems, 15(3):275– 298, 1998.
O. Buehler and J. Wegener. Evolutionary functional testing of an automated parking system. In International Conference on Computer, Communication and Control Technologies and The 9th. International Conference on Information Systems Analysis and Synthesis, Orlando, Florida, USA, 2003.
![Page 35: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/35.jpg)
Important Publications
Phil McMinn: Search-based software test data generation: a survey. Software Testing, Verification and Reliability 14(2): 105-156, 2004
Wasif Afzal, Richard Torkar and Robert Feldt: A Systematic Review of Search-based Testing for Non-Functional System Properties. Information and Software Technology, 51(6):957-976, 2009
Shaukat Ali, Lionel Briand, Hadi Hemmati and Rajwinder Panesar-Walawege: A Systematic Review of the Application and Empirical Investigation of Search-Based Test-Case Generation. IEEE Transactions on Software Engineering, To appear, 2010
M. Harman and B. Jones: Search-based software engineering. Information and Software Technology, 43(14):833–839, 2001.
M. Harman: The Current State and Future of Search Based Software Engineering, In Proceedings of the 29th International Conference on Software Engineering (ICSE 2007), 20-26 May, Minneapolis, USA (2007)
D. Whitley: An overview of evolutionary algorithms: Practical issues and common pitfalls. Information and Software Technology, 43(14):817–831, 2001.
O Bühler and J Wegener: Evolutionary functional testing, Computers & Operations Research, 2008 - Elsevier
J. Wegener and M. Grochtmann. Verifying timing constraints of real-time systems by means of evolutionary testing. Real-Time Systems, 15(3):275– 298, 1998.
O. Buehler and J. Wegener. Evolutionary functional testing of an automated parking system. In International Conference on Computer, Communication and Control Technologies and The 9th. International Conference on Information Systems Analysis and Synthesis, Orlando, Florida, USA, 2003.
Surveys
![Page 36: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/36.jpg)
Important Publications
Phil McMinn: Search-based software test data generation: a survey. Software Testing, Verification and Reliability 14(2): 105-156, 2004
Wasif Afzal, Richard Torkar and Robert Feldt: A Systematic Review of Search-based Testing for Non-Functional System Properties. Information and Software Technology, 51(6):957-976, 2009
Shaukat Ali, Lionel Briand, Hadi Hemmati and Rajwinder Panesar-Walawege: A Systematic Review of the Application and Empirical Investigation of Search-Based Test-Case Generation. IEEE Transactions on Software Engineering, To appear, 2010
M. Harman and B. Jones: Search-based software engineering. Information and Software Technology, 43(14):833–839, 2001.
M. Harman: The Current State and Future of Search Based Software Engineering, In Proceedings of the 29th International Conference on Software Engineering (ICSE 2007), 20-26 May, Minneapolis, USA (2007)
D. Whitley: An overview of evolutionary algorithms: Practical issues and common pitfalls. Information and Software Technology, 43(14):817–831, 2001.
O Bühler and J Wegener: Evolutionary functional testing, Computers & Operations Research, 2008 - Elsevier
J. Wegener and M. Grochtmann. Verifying timing constraints of real-time systems by means of evolutionary testing. Real-Time Systems, 15(3):275– 298, 1998.
O. Buehler and J. Wegener. Evolutionary functional testing of an automated parking system. In International Conference on Computer, Communication and Control Technologies and The 9th. International Conference on Information Systems Analysis and Synthesis, Orlando, Florida, USA, 2003.
Getting started in SBSE
![Page 37: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/37.jpg)
Important Publications
Phil McMinn: Search-based software test data generation: a survey. Software Testing, Verification and Reliability 14(2): 105-156, 2004
Wasif Afzal, Richard Torkar and Robert Feldt: A Systematic Review of Search-based Testing for Non-Functional System Properties. Information and Software Technology, 51(6):957-976, 2009
Shaukat Ali, Lionel Briand, Hadi Hemmati and Rajwinder Panesar-Walawege: A Systematic Review of the Application and Empirical Investigation of Search-Based Test-Case Generation. IEEE Transactions on Software Engineering, To appear, 2010
M. Harman and B. Jones: Search-based software engineering. Information and Software Technology, 43(14):833–839, 2001.
M. Harman: The Current State and Future of Search Based Software Engineering, In Proceedings of the 29th International Conference on Software Engineering (ICSE 2007), 20-26 May, Minneapolis, USA (2007)
D. Whitley: An overview of evolutionary algorithms: Practical issues and common pitfalls. Information and Software Technology, 43(14):817–831, 2001.
O Bühler and J Wegener: Evolutionary functional testing, Computers & Operations Research, 2008 - Elsevier
J. Wegener and M. Grochtmann. Verifying timing constraints of real-time systems by means of evolutionary testing. Real-Time Systems, 15(3):275– 298, 1998.
O. Buehler and J. Wegener. Evolutionary functional testing of an automated parking system. In International Conference on Computer, Communication and Control Technologies and The 9th. International Conference on Information Systems Analysis and Synthesis, Orlando, Florida, USA, 2003.
Applications
![Page 38: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/38.jpg)
Search-Based Structural Test Data
Generation
![Page 39: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/39.jpg)
Covering a structure
TARGET
![Page 40: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/40.jpg)
Fitness evaluation
TARGET
The test data executes the ‘wrong’ path
![Page 41: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/41.jpg)
Analysing control flow
TARGET
The outcomes at key decision statements
matter.
These are the decisions on
which the target is control dependent
![Page 42: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/42.jpg)
Approach Level
TARGET
= 2
= 1
= 0
minimisation
![Page 43: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/43.jpg)
Analysing predicatesApproach level alone gives us coarse values
a = 50, b = 0 a = 45, b = 5 a = 40, b = 10 a = 35, b = 15 a = 30, b = 20 a = 25, b = 25 !
getting ‘closer’ to being true
![Page 44: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/44.jpg)
Branch distanceAssociate a distance formula with different
relational predicates
a = 50, b = 0 branch distance = 50a = 45, b = 5 branch distance = 40a = 40, b = 10 branch distance = 30a = 35, b = 15 branch distance = 20a = 30, b = 20 branch distance = 10a = 25, b = 25 branch distance = 0
getting ‘closer’ to being true
![Page 45: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/45.jpg)
Branch distances for relational predicates
![Page 46: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/46.jpg)
Putting it all together
true
true
if a >= b
if b >= c
TARGET
TARGET MISSEDApproach Level = 1
Branch Distance = c - b
TARGET MISSEDApproach Level = 2
Branch Distance = b - a
false
false
true if c >= d false
TARGET MISSEDApproach Level = 0
Branch Distance = d - c
Fitness = approach Level + normalised branch distance
TARGET
normalised branch distance between 0 and 1indicates how close approach level is to being penetrated
![Page 47: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/47.jpg)
Normalisation FunctionsSince the ‘maximum’ branch distance is generally unknown we need a non-standard normalisation function
Baresel (2000), alpha = 1.001
![Page 48: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/48.jpg)
Normalisation FunctionsSince the ‘maximum’ branch distance is generally unknown we need a non-standard normalisation function
Arcuri (2010), beta = 1
![Page 49: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/49.jpg)
decrease decrease decrease
void fn( input1, input2, input3 .... )
increase increase increase
Alternating Variable Method
![Page 50: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/50.jpg)
Alternating Variable Method
Input variable value
Fitn
ess
Accelerated hill climb
![Page 51: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/51.jpg)
Alternating Variable Method1. Randomly generate start point
a=10, b=20, c=30
2. ‘Probe’ moves on a
a=9, b=20, c=30 no effect
a=11, b=20, c=30
3. ‘Probe’ moves on b
a=10, b=19, c=30
4. Accelerated moves in direction of improvement
improved
branch distance
![Page 52: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/52.jpg)
A search-based test data generator tool
![Page 53: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/53.jpg)
IGUANA(Java)
Test object
(C code compiled to a DLL )
inputs
information from test object
instrumentation
Java Native Interface
fitness computation
search algorithm
![Page 54: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/54.jpg)
A function for testing
![Page 55: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/55.jpg)
1. Parse the code and extract control dependency graph
TARGET
“which decisions are key for the execution of individual structural
targets” ?
if a == b
if b == c
return 1
Test Object Preparation
![Page 56: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/56.jpg)
2. Instrument the code
TARGET
if a == b
if b == c
return 1
for monitoring control flow and variable values in
predicates
Test Object Preparation
![Page 57: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/57.jpg)
3. Map inputs to a vector
Test Object Preparation
a b c
Straightforward in many cases
Inputs composed of dynamic data structures are harder to compose
Kiran Lakhotia, Mark Harman and Phil McMinn.Handling Dynamic Data Structures in Search-Based Testing.
Proceedings of the Genetic and Evolutionary Computation Conference (GECCO 2008), Atlanta, Georgia, USA, July 12-16 2008, pp. 1759-1766, ACM Press.
![Page 58: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/58.jpg)
Instrumentation
![Page 59: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/59.jpg)
Each branching condition is
replaced by a call to the function
node(...)
the instrumentation should only observe the program and not alter its behaviour
![Page 60: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/60.jpg)
The first parameter is the control flow
graph node ID of the decision statement
The second parameter is a boolean condition that replicates the structure in the original program
(i.e. including short-circuiting)
![Page 61: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/61.jpg)
Relational predicates are replaced with functions that compute branch distance.
![Page 62: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/62.jpg)
The instrumentation tells us:
Which decision nodes were executed
and their outcome (branch distances)
Therefore we can find which decision control flow diverged from a target for an input....
... and compute the approach level from the control dependence graph
... and lookup the branch distance
fitness value for an input
![Page 63: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/63.jpg)
TARGET
if a == b
if b == c
return 1
1nput: <20, 20, 30>NODE T F
1 0 1
2 10 0
4 20 0
....
Diverged at node 2approach level: 0
branch distance: 10
fitness = 0.009945219
![Page 64: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/64.jpg)
Which searchalgorithm?
![Page 65: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/65.jpg)
Empirical Study Bibclean Defroster F2 Eurocheck Gimp Space Spice Tiff Totinfo
Mark Harman and Phil McMinn.A Theoretical and Empirical Study of Search Based Testing: Local, Global and Hybrid Search.
IEEE Transactions on Software Engineering, 36(2), pp. 226-247, 2010.
760 branches in ~5 kLOC
![Page 66: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/66.jpg)
Interesting branches
20 98
Alternating Variable Method
Evolutionary Testing
![Page 67: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/67.jpg)
0
10
20
30
40
50
60
70
80
90
100
f2 F2 (
11F)
spac
e spa
ce_s
eqrot
rg (17
T)
spac
e spa
ce_s
eqrot
rg (22
T)
spice
clip_
to_cir
cle (4
9T)
spice
clip_
to_cir
cle (6
2T)
spice
clip_
to_cir
cle (6
8F)
spice
clipa
rc (22
T)
spice
clipa
rc (24
T)
spice
clipa
rc (63
F)
totinf
o Info
Tbl (14
F)
totinf
o Info
Tbl (21
F)
totinf
o Info
Tbl (29
T)
totinf
o Info
Tbl (29
F)
totinf
o Info
Tbl (35
T)
totinf
o Info
Tbl (35
F)
Suc
cess
Rat
e (%
)
Evolutionary Testing
Hill Climbing
Wins for the AVM
![Page 68: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/68.jpg)
1
10
100
1,000
10,000
100,000
gimp_
rgb_to
_hsl
(4T)
gimp_
rgb_to
_hsv
(5F)
gimp_
rgb_to
_hsv
4 (11
F)
gimp_
rgb_to
_hsv
_int (1
0T)
gradie
nt_ca
lc_bil
inear_
factor
(8T)
gradie
nt_ca
lc_co
nical_
asym
_facto
r (3F
)
gradie
nt_ca
lc_co
nical_
sym_fa
ctor (
3F)
gradie
nt_ca
lc_sp
iral_f
actor
(3F)
clipa
rc (13
F)
clipa
rc (15
T)
clipa
rc (15
F)
TIFF_SetS
ample
(5T)
Aver
gage
num
ber o
f fitn
ess
eval
uatio
ns
Evolutionary Testing
Hill Climbing
Wins for the AVM
![Page 69: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/69.jpg)
When does the AVM win?
fitne
ss
inputinput
![Page 70: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/70.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
chec
k_IS
BN (23F
)
chec
k_IS
BN (27T
)
chec
k_IS
BN (29T
)
chec
k_IS
BN (29F
)
chec
k_IS
SN (23F
)
chec
k_IS
SN (27T
)
chec
k_IS
SN (29T
)
chec
k_IS
SN (29F
)
Suc
cess
Rat
e
Evolutionary Testing
Hill Climbing
Wins for Evolutionary Testing
![Page 71: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/71.jpg)
When does ET win?
The branches in question were part of a routine for validating ISBN/ISSN strings
When a valid character is found, a counter variable is incremented
![Page 72: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/72.jpg)
When does ET win?Evolutionary algorithms incorporate
a population of candidate solutions
crossover
Crossover enables valid characters to be crossed over into different ISBN/ISSN strings
![Page 73: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/73.jpg)
Schemata
1010100011110000111010
1111101010000000101011
0001001010000111101011
![Page 74: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/74.jpg)
Schemata
1010100011110000111010
1111101010000000101011
0001001010000111101011
The schema theory predicts that schema of above average fitness will proliferate in subsequent
generations of the evolutionary search
![Page 75: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/75.jpg)
Schemata
1**1 *11*
1111
![Page 76: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/76.jpg)
The Genetic AlgorithmRoyal Road
S1: 1111****************************S2: ****1111************************S3: ********1111********************S4: ************1111****************S5: ****************1111************S6: ********************1111********S7: ************************1111****S8: ****************************1111S9: 11111111************************S10: ********11111111****************S11: ****************11111111********S12: ************************11111111S13: 1111111111111111****************S14: ****************1111111111111111S15: 11111111111111111111111111111111
![Page 77: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/77.jpg)
When Crossover Helps Executes the target
R1 R2
Q1
P1 P2
Q2
P3 P4
Q1
P1 P2
Q2
P3 P4
![Page 78: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/78.jpg)
When Crossover Helps Executes the target
Contains 2 valid characters
Contains1 valid
character
Contains1 valid
character
Contains1 valid
character
Contains1 valid
character
Contains1 valid
character
Contains1 valid
character
Contains1 valid
character
Contains1 valid
character
Contains 2 valid characters
Contains 2 valid characters
Contains 2 valid characters
Contains 4 valid characters
Contains 4 valid characters
![Page 79: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/79.jpg)
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
chec
k_IS
BN (23F
)
chec
k_IS
BN (27T
)
chec
k_IS
BN (29T
)
chec
k_IS
BN (29F
)
chec
k_IS
SN (23F
)
chec
k_IS
SN (27T
)
chec
k_IS
SN (29T
)
chec
k_IS
SN (29F
)
Suc
cess
Rat
e
Evolutionary Testing
Headless Chicken Test
Headless Chicken TestT. Jones, “Crossover, macromutation and population-based search,” Proc. ICGA ʼ95. Morgan Kaufmann, 1995, pp. 73–80.
![Page 80: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/80.jpg)
Investigations into Crossover
Royal Roads
HIFF
Real Royal Roads
Ignoble Trails
M. Mitchell, S. Forrest, and J. H. Holland, “The royal road for genetic algorithms: Fitness landscapes and GA performance,” Proc. 1st European Conference on Artificial Life. MIT Press, 1992, pp. 245–254.
R. A. Watson, G. S. Hornby, and J. B. Pollack, “Modeling building-block interdependency,” Proc. PPSN V. Springer, 1998, pp.97–106.
T. Jansen and I. Wegener, “Real royal road functions - where crossover provably is essential,” Discrete Applied Mathematics, vol. 149, pp. 111–125, 2005.
J. N. Richter, A. Wright, and J. Paxton, “Ignoble trails - where crossover is provably harmful,” Proc. PPSN X. Springer, 2008, pp. 92–101.
![Page 81: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/81.jpg)
Evolutionary Testing Schemata
{(a, b, c) | a = b}
{(a, b, c) | a > 0}
(50, 50, 25)(100, 100, 10)
...
(50, 10, 25)(100, -50, 10)
...
![Page 82: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/82.jpg)
Crossover of good schemata
{(a, b, c) | a = b} {(a, b, c) | b ≥ 100}
{(a, b, c) | a = b ∧ b ≥ 100}
subschema subschema
superschema
building block building block
![Page 83: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/83.jpg)
Crossover of good schemata
{(a, b, c) | a = b ∧ b ≥ 100} {(a, b, c) | c ≤ 10}
{(a, b, c) | a = b ∧ b ≥ 100 ∧ c ≤ 10} coveringschema
![Page 84: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/84.jpg)
What types of program and program structure enable Evolutionary Testing to perform well, through crossover and how?
P. McMinn, How Does Program Structure Impact the Effectiveness of the Crossover Operator in Evolutionary Testing? Proc. Symposium on Search-Based Software Engineering, 2010
1. Large numbers of conjuncts in the input condition
{(a, b, c...) | a = b ∧ b ≥ 100 ∧ c ≤ 10 ...
each represents a ‘sub’-test data generation problem that can be solved independently and
combined with other partial solutions
![Page 85: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/85.jpg)
P. McMinn, How Does Program Structure Impact the Effectiveness of the Crossover Operator in Evolutionary Testing? Proc. Symposium on Search-Based Software Engineering, 2010
2. Conjuncts should reference disjoint sets of variables
{(a, b, c, d ...) | a = b ∧ b = c ∧ c = d ...
the solution of each conjunct independently does not necessarily result in an overall solution
What types of program and program structure enable Evolutionary Testing to perform well, through crossover and how?
![Page 86: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/86.jpg)
Progressive Landscape
fitne
ss
input input
![Page 87: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/87.jpg)
Crossover - Conclusions
Crossover lends itself to programs/units that process large data structures (e.g. strings, arrays) resulting in input condition conjuncts with disjoint variables
1. Large numbers of conjuncts in the input condition 2. Conjuncts should reference disjoint sets of variables
... or units that require large sequences of method calls to move an object into a required state
e.g. testing for a full stack - push(...), push(...), push(...)
![Page 88: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/88.jpg)
Other Theoretical Work
A. Arcuri, P. K. Lehre, and X. Yao, “Theoretical runtime analyses of search algorithms on the test data generation for the triangle classification problem,” SBST workshop 2008, Proc. ICST 2008. IEEE, 2008, pp. 161–169.
A. Arcuri, “Longer is better: On the role of test sequence length in software testing,” Proc. ICST 2010. IEEE, 2010.
![Page 89: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/89.jpg)
TestabilityTransformation
![Page 90: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/90.jpg)
The ‘Flag’ Problem
![Page 91: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/91.jpg)
Program Transformation
![Page 92: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/92.jpg)
Testability transformation: change the program to improve test data generation
... whilst preserving test adequacy
Programs will inevitably have features that heuristic searches handle less well
Mark Harman, Lin Hu, Rob Hierons, Joachim Wegener, Harmen Sthamer, Andre Baresel and Marc Roper. Testability Transformation.
IEEE Transactions on Software Engineering. 30(1): 3-16, 2004.
![Page 93: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/93.jpg)
Nesting
Phil McMinn, David Binkley and Mark Harman Empirical Evaluation of a Nesting Testability Transformation for Evolutionary Testing
ACM Transactions on Software Engineering and Methodology, 18(3), Article 11, May 2009
![Page 94: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/94.jpg)
Testability Transformation
Note that the programs are no longer equivalent
But we don’t care - so long as we get the test data is still adequate
![Page 95: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/95.jpg)
Nesting & Local Optima
![Page 96: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/96.jpg)
Nesting & Local Optima
![Page 97: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/97.jpg)
-100
-80
-60
-40
-20
0
20
40
60
80
100
Nested branches
Chan
ge in
suc
cess
rate
afte
r app
lyin
g tr
ansf
orm
atio
n (%
)
Results - Industrial & Open source code
![Page 98: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/98.jpg)
Dependent & Independent Predicates
Predicates influenced by disjoint sets of input variables
IndependentCan be optimised
in parallel
e.g. ‘a == 0’ and ‘b == 0’
![Page 99: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/99.jpg)
Dependent & Independent Predicates
Predicates influenced by non-disjoint sets of input
variables
DependentInteractions
between predicates inhibit
parallel optimisation
e.g. ‘a == b’ and ‘b == c’
![Page 100: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/100.jpg)
-100
-80
-60
-40
-20
0
20
40
60
80
100
Nested branches
Chan
ge in
suc
cess
rate
afte
r app
lyin
g tr
ansf
orm
atio
n (%
)
Dependent predicates
![Page 101: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/101.jpg)
-100
-80
-60
-40
-20
0
20
40
60
80
100
Nested branches
Chan
ge in
suc
cess
rate
afte
r app
lyin
g tr
ansf
orm
atio
n (%
)
Dependent predicates
Independent and some dependent predicates
![Page 102: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/102.jpg)
When not preserving program equivalence can go wrong
![Page 103: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/103.jpg)
so we need to be careful:are we still testing according to the same
criterion?
we are testing to cover structure
... but the structure is the problem
so we transform the program
... but this alters the structure
![Page 104: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/104.jpg)
Input Domain Reduction
![Page 105: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/105.jpg)
Mark Harman, Youssef Hassoun, Kiran Lakhotia, Phil McMinn and Joachim Wegener.The Impact of Input Domain Reduction on Search-Based Test Data Generation.
The 6th joint meeting of the European Software Engineering Conference and the ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/
FSE 2007), Cavtat, Croatia, September 3-7 2007, pp. 155-164, ACM Press.
![Page 106: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/106.jpg)
Effect of Reduction-100,000 ... 100,000-100,000 ... 100,000
-100,000 ... 100,000
approx 1016
![Page 107: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/107.jpg)
Effect of Reduction-100,000 ... 100,000-100,000 ... 100,000
-100,000 ... 100,000
200,001
![Page 108: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/108.jpg)
Variable Dependency Analysis
![Page 109: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/109.jpg)
Empirical Study
Random Search Alternating Variable Method Evolutionary Testing
Defroster F2
Gimp Spice
Tiff
Case studies:
Studied the effects of reduction with:
![Page 110: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/110.jpg)
Effect on Random Testing-49 ... 50
Probability of executing target:
-49 ... 50-49 ... 50
100 x 100 x 1
100 x 100 x 100
![Page 111: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/111.jpg)
Results with Random Testing
![Page 112: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/112.jpg)
Results with AVM
![Page 113: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/113.jpg)
Effect on AVMSaves probe moves (and thus wasteful fitness evaluations) around irrelevant variables
void fn( irrelevant1, irrelevant2, irrelevant3, required1 .... )
increase increase increase increase
decrease decrease decrease decrease
![Page 114: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/114.jpg)
Effect on AVMSaves probe moves (and thus wasteful fitness evaluations) around irrelevant variables
void fn( irrelevant1, irrelevant2, irrelevant3, required1 .... )
increase
decrease
![Page 115: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/115.jpg)
Effect on ET
Saves mutations on irrelevant variables
Mutations concentrated on the variables that matter
Likely to speed up the search
![Page 116: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/116.jpg)
Results with ET
![Page 117: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/117.jpg)
Conclusions for Input Domain Reduction
Variable dependency analysis can be used to reduce input domains
This can reduce search effort for the AVM and ET
Perhaps surprisingly, there is no overall change for random search
![Page 118: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/118.jpg)
Other applications of Search-Based Testing
Mutation testing
State machine testing
Y. Jia and M. Harman. Constructing subtle faults using higher order mutation testing. In 8th International Working Conference on Source Code Analysis and Manipulation (SCAM 2008,
Beijing, China, 2008. IEEE Computer Society. To appear.
K. Derderian, R. Hierons, M. Harman, and Q. Guo. Automated unique input output sequence generation for conformance testing of FSMs. The Computer Journal,
39:331–344, 2006.
![Page 119: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/119.jpg)
Other applications of Search-Based Testing
Test suite reductionPareto Efficient Multi-Objective Test Case Selection, Shin Yoo and Mark Harman.
Proceedings of ACM International Symposium on Software Testing and Analysis 2007 (ISSTA 2007), p140-150
(Time-aware) Test suite prioritisationK. R. Walcott, M. L. Soffa, G. M. Kapfhammer, and R. S. Roos. Time aware test suite prioritization. In ISSTA ’06: Proceedings of the 2006 international symposium on Software testing and analysis, pages 1–12, New York, NY, USA, 2006. ACM Press.
![Page 120: search based testing - for webtarot2010.ist.tugraz.at/slides/McMinn.pdf · Overview How and why Search-Based Testing Works Examples: Temporal, Functional, Structural Search-Based](https://reader033.vdocument.in/reader033/viewer/2022053106/6072011a22da0d4cac0c3074/html5/thumbnails/120.jpg)
Other applications of Search-Based Testing
Combinatorial Interaction Testing
GUI Testing
M.B. Cohen, M.B. Dwyer and J. Shi, Interaction testing of highly-configurable systems in the presence of constraints, International Symposium on Software Testing and Analysis
(ISSTA), London, July 2007, pp. 129-139.
X. Yuan, M.B. Cohen and A.M. Memon, GUI Interaction Testing: Incorporating Event Context, IEEE Transactions on Software Engineering, to appear