kyle mundt february 3, 2010. richard lipton, 1971 a way of testing your tests alter your code in...
TRANSCRIPT
![Page 1: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/1.jpg)
Mutation-Based Testing
Kyle MundtFebruary 3, 2010
![Page 2: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/2.jpg)
Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered code Becoming popular again due to increased
computing power Strongest use seems to be with Fortran and
C (procedural languages)
What is Mutation-Based Testing?
![Page 3: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/3.jpg)
Mutant: code resulting from applying a mutation operator
Mutation (or Fault) Operator: rule used to create a mutant
Killing a Mutant: when tests fail on a mutant Mutation-Adequate: tests kill all mutants Equivalent mutant: produces same output
as original (can’t be killed) Mutation score: killed / (total – equivalent)
Terms
![Page 4: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/4.jpg)
Original
if (a == 10)b = 3;
elseb = 5;
Example
Mutant
if (a != 10) b = 3;
else b = 5;
![Page 5: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/5.jpg)
Apply mutation operators to code (only one change in each mutant)
Run mutants through test cases Check mutants that do not fail
◦ Equivalent mutants Change test cases to catch mutant Repeat
The Process
![Page 6: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/6.jpg)
Don’t begin too early Code should already be written Tests should be reasonably thorough Begin with good set of test cases and code
that passes tests Iterative process
◦ Create mutants, run tests, fix, repeat
When to Perform?
![Page 7: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/7.jpg)
Traditional Mutation Operators◦ Statement deletion◦ Invert logical operators◦ Replace arithmetic operators◦ Replace variable with another in same scope
Class-Level Mutation Operators◦ Object oriented◦ Change containers◦ Concurrency
Examples of Mutation Operators
![Page 8: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/8.jpg)
If test results are the same◦ There is “dead code”◦ Mutant is equivalent to original◦ Test cases not complete
Code coverage not good enough Functional testing not enough Need way to analyze test effectiveness Useful for hardware verification Based on coupling effect: “ test data set that
detects all simple faults in a program is so sensitive that it also detects more complex faults”
Ties into automatic test generation
Why is it Important?
![Page 9: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/9.jpg)
Hitting every line doesn’t guarantee good tests
Doesn’t ensure defects detected if they occur
Not good measure of verification effectiveness
Code Coverage
![Page 10: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/10.jpg)
Testing functionality of program Deals with how program should work Mostly ignores how it shouldn’t Functional tests are subjective Also bad measure of verification quality
Functional Testing
![Page 11: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/11.jpg)
Looking at, improving tests Convinces us tests will detect defects if they
occur Similar to techniques done manually on
hardware Need way to automate it to be practical
Analyzing Tests
![Page 12: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/12.jpg)
Verify testbed Gives data to be used in improving chip
testing Hardware defects can cause unexpected
behavior Must ensure this behavior is caught Attempt to show defects found by tests
Hardware verification
![Page 13: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/13.jpg)
Adequate automation Huge (maybe infinite) possible number of
mutants Fault classification and prioritization Which operators to use How to apply to OOP, concurrency, etc.
Problems
![Page 14: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/14.jpg)
Mothra – 22 operators 6 operators account for 40-60% of mutants Redundant mutants (can be killed by same
test) Research shows 5 operators gives good
data◦ Gives 99% total mutant score and reduces
number of mutants by 77%
Selective Mutation
![Page 15: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/15.jpg)
ABS – make value of each arithmetic expression be 0, positive, and negative
AOR – replace arithmetic operator with valid operators
LCR – replace each logical operator ROR – replace relational operators UOI – insert unary operators in front of
expressions Note this is used on Fortran, a procedural
language
Five Operators
![Page 16: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/16.jpg)
Running whole program takes time Don’t really care about what happens after mutant
line hit So, don’t execute whole mutant Compare states of original and mutant after
mutated line executed Regular mutation requires three things
◦ Mutant be reached◦ Mutant creates incorrect state in program◦ Program state reflected as difference in test output
Weak Mutation only requires the first two Related to automatic test generation
Weak Mutation
![Page 17: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/17.jpg)
Put all mutants into “metaprogram” One large program to compile and link Mothra turns program into intermediate
form Modifies intermediate form to create
mutants Interprets intermediate forms (interpreting
is slow) Schema-based is much faster than
interpretive systems
Schema-Based Mutation
![Page 18: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/18.jpg)
Can reduce number of test cases executed◦ Once a test case is killed, remove it from further
testing◦ No reason to kill it twice
Random selection of mutants appropriate in some cases
Other Optimizations
![Page 19: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/19.jpg)
Easy to come up with operators for simple types
User types much harder Operators have been developed for things
like◦ Inheritance◦ Polymorphism◦ Method visibility
Harder to mutate based on meaning of object
Some research applied to Java OO mutation
OO Mutation
![Page 20: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/20.jpg)
Methods proposed to alter at code level◦ Allows changes to things like attributes
Mutants made this way may not compile Could cause integration errors Still doesn’t look at meanings Can write operators specifically for classes Probably not practical in almost all
situations How to actually change object state?
Changing Object State
![Page 21: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/21.jpg)
Mutate commonly used Java libraries Again operators picked to try and simulate
common mistakes One solution found was to mutate
Containers, Iterators, and InputStream Java reflection can also be used to examine
object fields at runtime
One Approach
![Page 22: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/22.jpg)
Collection Interface (most apply to List or Vector as well)◦ Make collection empty◦ Remove some element (first, last, random)◦ Reorder elements◦ Mutate element type
Iterators:◦ Skip element
InputStream◦ Skip bytes of data
Some Suggested Operators
![Page 23: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/23.jpg)
First described for C programs Designed for integration testing, mutate
interface between modules Designed to scale to larger systems Actions taken to control number of mutants
◦ Only look at integration errors◦ Tests only connections between pairs of
subsystems◦ Mutates only module interfaces (eg. Function
calls, return values)
Interface Mutation
![Page 24: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/24.jpg)
MuClipse◦ Open source mutation testing plug-in for
Eclipse µJava (muJava) Heckle (Ruby) Insure++ (C++) Nester (C#)
Implementations
![Page 25: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/25.jpg)
C# programs for Visual Studios 2005 Only supports NUnit Framework Highlights code
◦ Killed mutations in green◦ Surviving mutations in red◦ Code not covered by tests in blue
Allows use of XML-based grammar to define own transformation rules
Nester
![Page 26: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/26.jpg)
Different approach Used to find bugs in source code Creates functionally equivalent mutants
◦ Lines changed without changing expected results Tests should all still pass If a test fails, something wrong with code Insure++ reports faults and lines
responsible
Insure++
![Page 27: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/27.jpg)
More complicated programs equals greater need for quality tests
Need way to ensure/measure test effectiveness
Mutation-Based testing can fill this role Still needs further research Not a lot of implementations or examples of
use on large scale Getting attention in hardware verification More work needs to be done to apply to OO
Conclusion
![Page 28: Kyle Mundt February 3, 2010. Richard Lipton, 1971 A way of testing your tests Alter your code in various ways Check to see if tests fail on altered](https://reader035.vdocument.in/reader035/viewer/2022062718/56649e635503460f94b5f623/html5/thumbnails/28.jpg)
[1] Offutt, J. A. & Untch, R. H. (October 2000). Mutation 2000: Uniting the Orthogonal. Retrieved January 18, 2011 from http://cs.gmu.edu/~offutt/rsrch/papers/mut00.pdf
[2] Bakewell, G. (2010). Mutation-Based Testing Technologies Close the “Quality Gap” in Functional Verification for Complex Chip Designs. Retrieved January 18, 2011, from http://electronicdesign.com/article/eda/Mutation-Based-Testing-Technologies-Close-the-Quality-Gap-in-Functional-Verification-for-Complex-Chip-Designs/4.aspx
[3] Offut, J. A. (June 1995). A Practical System for Mutation Testing: Help for the Common Programmer . Retrieved January 18, 2011 from http://cs.gmu.edu/~offutt/rsrch/papers/practical.pdf
[4] Usaola, M. & Mateo, P. (2010). Mutation Testing Cost Reduction Techniques: A Survey. IEEE Software, 27(3), 80-86. Retrieved January 23, 2011, from ABI/INFORM Global. (Document ID: 2012103051).
[5] Kolawa, Adam (1999). Mutation Testing: A New Approach to Automatic Error-Detection. Retrieved January 18, 2011, from http://www.parasoft.com/jsp/products/article.jsp?articleId=291
[6] Alexander, R. T.; Bieman, J. M.; Ghosh, S.; & Ji, B (2002). Mutation of Java Objects. Retrieved January 18, 2011 from http://www.cs.colostate.edu/~bieman/Pubs/AlexanderBiemanGhoshJiISSRE02.pdf
[7] Nester - Free Software that Helps to do Effective Unit Testing in C#. http://nester.sourceforge.net/
Sources