better code through making bugs
DESCRIPTION
Do you know how good your tests are? Mutation testing can tell you. Unlike test coverage metrics (which only tell us how much of your application was executed, not whether the tests were any use), mutation testing lets us say something concrete about the quality of your test suite. Mutation testing has been around for years, but it's only recently that performance tools (such as PI Test for Java) have become available. In this session, Seb will look at the motivation and technology behind mutation testing and see some examples in action.TRANSCRIPT
![Page 1: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/1.jpg)
Tuesday, 22 April 14
![Page 2: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/2.jpg)
Be#er code through making bugs
Seb Rose
Claysnow Limited
@sebrose
Tuesday, 22 April 14
![Page 3: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/3.jpg)
Massive thanks to:
3
Henry Colespitest
Filip van Laenenmutant
Tuesday, 22 April 14
![Page 4: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/4.jpg)
How do you assure the quality of your test suite?
Tuesday, 22 April 14
![Page 5: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/5.jpg)
Only employ coding gods
How do you assure the quality of your test suite?
Tuesday, 22 April 14
![Page 6: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/6.jpg)
Only employ coding gods
TDD will protect us
How do you assure the quality of your test suite?
Tuesday, 22 April 14
![Page 7: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/7.jpg)
Only employ coding gods
TDD will protect us
Peer review
How do you assure the quality of your test suite?
Tuesday, 22 April 14
![Page 8: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/8.jpg)
Only employ coding gods
TDD will protect us
Peer review
QA will catch anything that we miss
How do you assure the quality of your test suite?
Tuesday, 22 April 14
![Page 9: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/9.jpg)
Only employ coding gods
TDD will protect us
Peer review
QA will catch anything that we miss
Good code coverage
How do you assure the quality of your test suite?
Tuesday, 22 April 14
![Page 10: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/10.jpg)
How do you assure the quality of your test suite?
Tuesday, 22 April 14
![Page 11: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/11.jpg)
Code coverage?
How do you assure the quality of your test suite?
Tuesday, 22 April 14
![Page 12: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/12.jpg)
Code coverage?
Line coverage?
How do you assure the quality of your test suite?
Tuesday, 22 April 14
![Page 13: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/13.jpg)
Code coverage?
Line coverage?Branch coverage?
How do you assure the quality of your test suite?
Tuesday, 22 April 14
![Page 14: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/14.jpg)
Code coverage?
Line coverage?Branch coverage?
Statement coverage?
How do you assure the quality of your test suite?
Tuesday, 22 April 14
![Page 15: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/15.jpg)
Does good coverage guarantee that:
Tuesday, 22 April 14
![Page 16: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/16.jpg)
• I can safely refactor my tests?
Does good coverage guarantee that:
Tuesday, 22 April 14
![Page 17: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/17.jpg)
• I can safely refactor my tests?
• I can trust a test suite I inherited?
Does good coverage guarantee that:
Tuesday, 22 April 14
![Page 18: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/18.jpg)
• I can safely refactor my tests?
• I can trust a test suite I inherited?
• My team are writing effective tests?
Does good coverage guarantee that:
Tuesday, 22 April 14
![Page 19: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/19.jpg)
• I can safely refactor my tests?
• I can trust a test suite I inherited?
• My team are writing effective tests?
• I've retrofitted enough tests to protect my legacy code?
Does good coverage guarantee that:
Tuesday, 22 April 14
![Page 20: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/20.jpg)
• I can safely refactor my tests?
• I can trust a test suite I inherited?
• My team are writing effective tests?
• I've retrofitted enough tests to protect my legacy code?
Does good coverage guarantee that:
NO, IT
DOESN’T
Tuesday, 22 April 14
![Page 21: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/21.jpg)
Coverage tells us nothing about the quality of our tests
Tuesday, 22 April 14
![Page 22: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/22.jpg)
Coverage tells us nothing about the quality of our tests
It tells us which statements have been run by our tests
Tuesday, 22 April 14
![Page 23: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/23.jpg)
Coverage tells us nothing about the quality of our tests
It tells us which statements have NOT been tested
Tuesday, 22 April 14
![Page 24: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/24.jpg)
Coverage tells us nothing about the quality of our tests
... but it is very useful for something else
Tuesday, 22 April 14
![Page 25: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/25.jpg)
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
![Page 26: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/26.jpg)
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
![Page 27: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/27.jpg)
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
![Page 28: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/28.jpg)
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
![Page 29: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/29.jpg)
@Test public void shouldStartWithEmptyCount() { assertEquals(0,testee.currentCount()); }
@Test public void shouldCountIntegersAboveTen() { testee.count(11); assertEquals(1,testee.currentCount()); } @Test public void shouldNotCountIntegersBelowTen() { testee.count(9); assertEquals(0,testee.currentCount()); }
Tuesday, 22 April 14
![Page 30: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/30.jpg)
Lipton, “Fault diagnosis in computer programs”,1971
“If we want to know if a test suite has properly checked some code
deliberately introduce a bug”
Tuesday, 22 April 14
![Page 31: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/31.jpg)
The deliberate introduction of a bug by changing the code under test
Mutation:
Tuesday, 22 April 14
![Page 32: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/32.jpg)
A version of the code under test that has had a single mutation
Mutant:
Tuesday, 22 April 14
![Page 33: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/33.jpg)
Tuesday, 22 April 14
![Page 34: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/34.jpg)
Mutation test:
Tuesday, 22 April 14
![Page 35: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/35.jpg)
Run your test suite
Mutation test:
Tuesday, 22 April 14
![Page 36: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/36.jpg)
Run your test suite
If any test fails, the mutant has been killed
Mutation test:
Tuesday, 22 April 14
![Page 37: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/37.jpg)
Run your test suite
If any test fails, the mutant has been killed
If no test fails, the mutant has survived
Mutation test:
Tuesday, 22 April 14
![Page 38: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/38.jpg)
Mutation testing:
Tuesday, 22 April 14
![Page 39: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/39.jpg)
Generate lots of mutants
Mutation testing:
Tuesday, 22 April 14
![Page 40: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/40.jpg)
Generate lots of mutants
Run each one through your test suite
Mutation testing:
Tuesday, 22 April 14
![Page 41: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/41.jpg)
Generate lots of mutants
Run each one through your test suite
Record and interpret the results
Mutation testing:
Tuesday, 22 April 14
![Page 42: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/42.jpg)
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
![Page 43: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/43.jpg)
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
if ( i > 10 ) {
Tuesday, 22 April 14
![Page 45: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/45.jpg)
Mutation operators:
• Conditionals Boundary Mutator• Negate Conditionals Mutator• Remove Conditionals Mutator• Math Mutator• Increments Mutator• Invert Negatives Mutator• Inline Constant Mutator• Return Values Mutator• Void Method Calls Mutator• Non Void Method Calls Mutator• Constructor Calls Mutator
Tuesday, 22 April 14
![Page 46: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/46.jpg)
“Generated mutants are similar to real faults”
Andrews, Briand, Labiche, ICSE 2005
Tuesday, 22 April 14
![Page 47: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/47.jpg)
“In practice, if the software contains a fault, there will usually be a set of mutants that can only be killed by a test case that also detects that fault.”
Geist et. al., “Estimation and Enhancement of Real-time Software Reliability through Mutation Analysis,” 1992
Tuesday, 22 April 14
![Page 48: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/48.jpg)
“Complex faults are coupled to simple faults in such a way that a test data set that detects all simple faults in a program will detect most complex faults.”
K. Wah, “Fault Coupling in Finite Bijective Functions,” 1995
Tuesday, 22 April 14
![Page 49: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/49.jpg)
Poor performance
Equivalent mutations
Why isn’t mutation testing widely used?
Tuesday, 22 April 14
![Page 50: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/50.jpg)
How bad is performance?
Tuesday, 22 April 14
![Page 51: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/51.jpg)
How bad is performance?
Joda Time, consider, let us
Tuesday, 22 April 14
![Page 52: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/52.jpg)
Joda Time is a ...
Tuesday, 22 April 14
![Page 53: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/53.jpg)
small library for dealing with dates and times
Joda Time is a ...
Tuesday, 22 April 14
![Page 54: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/54.jpg)
small library for dealing with dates and times
68k lines of code
Joda Time is a ...
Tuesday, 22 April 14
![Page 55: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/55.jpg)
small library for dealing with dates and times
68k lines of code70k lines of test code
Joda Time is a ...
Tuesday, 22 April 14
![Page 56: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/56.jpg)
small library for dealing with dates and times
68k lines of code70k lines of test code
Takes about 10 seconds to compile
Joda Time is a ...
Tuesday, 22 April 14
![Page 57: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/57.jpg)
small library for dealing with dates and times
68k lines of code70k lines of test code
Takes about 10 seconds to compileTakes about 16 seconds to run the unit tests
Joda Time is a ...
Tuesday, 22 April 14
![Page 58: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/58.jpg)
Tuesday, 22 April 14
![Page 59: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/59.jpg)
Let’s use 10 mutation operators
Tuesday, 22 April 14
![Page 60: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/60.jpg)
Let’s use 10 mutation operatorsassume about 10k mutations
Tuesday, 22 April 14
![Page 61: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/61.jpg)
Let’s use 10 mutation operatorsassume about 10k mutations
If it takes 1 second to compile each one
Tuesday, 22 April 14
![Page 62: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/62.jpg)
Let’s use 10 mutation operatorsassume about 10k mutations
If it takes 1 second to compile each one 2.5 hours to generate the mutants
Tuesday, 22 April 14
![Page 63: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/63.jpg)
Let’s use 10 mutation operatorsassume about 10k mutations
If it takes 1 second to compile each one 2.5 hours to generate the mutants
Run the test suite for each mutant
Tuesday, 22 April 14
![Page 64: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/64.jpg)
Let’s use 10 mutation operatorsassume about 10k mutations
If it takes 1 second to compile each one 2.5 hours to generate the mutants
Run the test suite for each mutant10k x 16 seconds = 44.5 hours
Tuesday, 22 April 14
![Page 65: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/65.jpg)
pitest manipulates the byte code directly
10k mutants generated < 1 second
Don’t compile!
Tuesday, 22 April 14
![Page 66: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/66.jpg)
Run fewer tests!
Tuesday, 22 April 14
![Page 67: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/67.jpg)
Stop when a test fails
Run fewer tests!
Tuesday, 22 April 14
![Page 68: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/68.jpg)
Stop when a test failscan easily halve the run time
Run fewer tests!
Tuesday, 22 April 14
![Page 69: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/69.jpg)
Stop when a test failscan easily halve the run time
Choose your tests carefully
Run fewer tests!
Tuesday, 22 April 14
![Page 70: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/70.jpg)
Stop when a test failscan easily halve the run time
Choose your tests carefullynot every test can kill every mutant
Run fewer tests!
Tuesday, 22 April 14
![Page 71: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/71.jpg)
Stop when a test failscan easily halve the run time
Choose your tests carefullynot every test can kill every mutant
Parallelise the test runner
Run fewer tests!
Tuesday, 22 April 14
![Page 72: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/72.jpg)
Stop when a test failscan easily halve the run time
Choose your tests carefullynot every test can kill every mutant
Parallelise the test runneryour tests are unit tests, right?
Run fewer tests!
Tuesday, 22 April 14
![Page 73: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/73.jpg)
Choosing your tests well is critical
Tuesday, 22 April 14
![Page 74: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/74.jpg)
Each mutant can only ever be killed
Choosing your tests well is critical
Tuesday, 22 April 14
![Page 75: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/75.jpg)
Each mutant can only ever be killed
by a subset of the tests
Choosing your tests well is critical
Tuesday, 22 April 14
![Page 76: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/76.jpg)
Each mutant can only ever be killed
by a subset of the tests
Every other test run is waste
Choosing your tests well is critical
Tuesday, 22 April 14
![Page 77: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/77.jpg)
How can we know which tests might kill
any given mutant?
Tuesday, 22 April 14
![Page 78: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/78.jpg)
How can we know which tests might kill
any given mutant?
Naming conventions?
Tuesday, 22 April 14
![Page 79: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/79.jpg)
How can we know which tests might kill
any given mutant?
Naming conventions?Static analysis?
Tuesday, 22 April 14
![Page 80: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/80.jpg)
How can we know which tests might kill
any given mutant?
Naming conventions?Static analysis?
COVERAGE DATA!
Tuesday, 22 April 14
![Page 81: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/81.jpg)
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
![Page 82: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/82.jpg)
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
![Page 83: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/83.jpg)
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
![Page 84: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/84.jpg)
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
Tuesday, 22 April 14
![Page 85: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/85.jpg)
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
shouldCountIntegersAboveTenshouldNotCountIntegersBelowTenshouldCountIntegersOfExactlyTen
Tuesday, 22 April 14
![Page 86: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/86.jpg)
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
shouldCountIntegersAboveTenshouldNotCountIntegersBelowTenshouldCountIntegersOfExactlyTen
shouldCountIntegersAboveTenshouldCountIntegersOfExactlyTen
Tuesday, 22 April 14
![Page 87: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/87.jpg)
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
shouldCountIntegersAboveTenshouldNotCountIntegersBelowTenshouldCountIntegersOfExactlyTen
shouldCountIntegersAboveTenshouldCountIntegersOfExactlyTen
not covered by any test
Tuesday, 22 April 14
![Page 88: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/88.jpg)
public class CountingClass { private int count; public void count(int i) { if ( i >= 10 ) { count++; } } public void reset() { count = 0; }}
shouldCountIntegersAboveTenshouldNotCountIntegersBelowTenshouldCountIntegersOfExactlyTen
shouldCountIntegersAboveTenshouldCountIntegersOfExactlyTen
not covered by any test
shouldStartWithEmptyCount does not cover any of the lines shown
Tuesday, 22 April 14
![Page 89: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/89.jpg)
Joda Time on my machine, with 2 threads:
- Timings> scan classpath : < 1 second> coverage and dependency analysis : 59 seconds> build mutation tests : 1 seconds> run mutation analysis : 8 minutes and 21 seconds> Total : 9 minutes and 22 seconds
- Statistics>> Generated 9922 mutations Killed 7833 (79%)>> Ran 117579 tests (11.85 tests per mutation)
Tuesday, 22 April 14
![Page 90: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/90.jpg)
Why only 79% killed?
Tuesday, 22 April 14
![Page 91: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/91.jpg)
Missing test cases
Why only 79% killed?
Tuesday, 22 April 14
![Page 92: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/92.jpg)
Missing test cases
Time outs
Why only 79% killed?
Tuesday, 22 April 14
![Page 93: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/93.jpg)
Missing test cases
Time outs
Equivalent mutations
Why only 79% killed?
Tuesday, 22 April 14
![Page 94: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/94.jpg)
public void someLogic(int i) { if (i <= 100) { throw new IllegalArgumentException(); }
if (i >= 100) { doSomething(); }}
Equivalent mutant:
Tuesday, 22 April 14
![Page 95: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/95.jpg)
public void someLogic(int i) { if (i <= 100) { throw new IllegalArgumentException(); }
if (i >= 100) { doSomething(); }}
if (i > 100) {
Equivalent mutant:
Tuesday, 22 April 14
![Page 96: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/96.jpg)
public void someLogic(int i) { if (i <= 100) { throw new IllegalArgumentException(); }
if (i >= 100) { doSomething(); }}
if (i > 100) {
i can never be 100 here
Equivalent mutant:
Tuesday, 22 April 14
![Page 98: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/98.jpg)
Maybe we should have written:
public void someLogic(int i) { if (i <= 100) { throw new IllegalArgumentException(); }
doSomething();}
Tuesday, 22 April 14
![Page 99: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/99.jpg)
Some causes of equivalence are:
Tuesday, 22 April 14
![Page 100: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/100.jpg)
Dead/useless code
Some causes of equivalence are:
Tuesday, 22 April 14
![Page 101: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/101.jpg)
Dead/useless codeNon-functional modifications
Some causes of equivalence are:
Tuesday, 22 April 14
![Page 102: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/102.jpg)
Dead/useless codeNon-functional modificationsUnsatisfiable guards
Some causes of equivalence are:
Tuesday, 22 April 14
![Page 103: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/103.jpg)
Dead/useless codeNon-functional modificationsUnsatisfiable guardsInternal state
Some causes of equivalence are:
Tuesday, 22 April 14
![Page 104: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/104.jpg)
Dead/useless codeNon-functional modificationsUnsatisfiable guardsInternal state
Some causes of equivalence are:
This can help us improve our code
Tuesday, 22 April 14
![Page 105: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/105.jpg)
Mutants that survive need to bechecked manually
to determine if they are equivalent
Tuesday, 22 April 14
![Page 106: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/106.jpg)
Is this a show-stopper?
Tuesday, 22 April 14
![Page 107: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/107.jpg)
Is this a show-stopper?
Not all that common in practice
Tuesday, 22 April 14
![Page 108: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/108.jpg)
Is this a show-stopper?
Not all that common in practice
TDD helps prevent equivalents
Tuesday, 22 April 14
![Page 109: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/109.jpg)
Is this a show-stopper?
Not all that common in practice
TDD helps prevent equivalents
Tolerate a few survivors
Tuesday, 22 April 14
![Page 110: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/110.jpg)
What about really large code bases?
Tuesday, 22 April 14
![Page 111: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/111.jpg)
What about really large code bases?
Mutation testing can take a long time:
Tuesday, 22 April 14
![Page 112: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/112.jpg)
What about really large code bases?
Mutation testing can take a long time:Use fewer mutants
Tuesday, 22 April 14
![Page 113: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/113.jpg)
What about really large code bases?
Mutation testing can take a long time:Use fewer mutantsFilter the candidates
Tuesday, 22 April 14
![Page 114: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/114.jpg)
What about really large code bases?
Mutation testing can take a long time:Use fewer mutantsFilter the candidatesRun infrequently
Tuesday, 22 April 14
![Page 115: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/115.jpg)
Get the developers to mutation test
Tuesday, 22 April 14
![Page 116: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/116.jpg)
Get the developers to mutation test
Only on code they are working on
Tuesday, 22 April 14
![Page 117: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/117.jpg)
Get the developers to mutation test
Only on code they are working on
Faster feedback
Tuesday, 22 April 14
![Page 118: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/118.jpg)
Get the developers to mutation test
Only on code they are working on
Faster feedback
Improves design of the code
Tuesday, 22 April 14
![Page 119: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/119.jpg)
Get your CI server to run the mutation test
Tuesday, 22 April 14
![Page 120: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/120.jpg)
Get your CI server to run the mutation test
Over whole codebase if quick enough
Tuesday, 22 April 14
![Page 121: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/121.jpg)
Get your CI server to run the mutation test
Over whole codebase if quick enough
Use SCM integration to identify subset
Tuesday, 22 April 14
![Page 122: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/122.jpg)
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
![Page 123: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/123.jpg)
Use mutation testing from day 1
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
![Page 124: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/124.jpg)
Use mutation testing from day 1 Start on a small code base
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
![Page 125: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/125.jpg)
Use mutation testing from day 1 Start on a small code baseKeep number of unit tests per class low
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
![Page 126: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/126.jpg)
Use mutation testing from day 1 Start on a small code baseKeep number of unit tests per class lowHave small classes
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
![Page 127: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/127.jpg)
Use mutation testing from day 1 Start on a small code baseKeep number of unit tests per class lowHave small classesSelect a good tool:
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
![Page 128: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/128.jpg)
Use mutation testing from day 1 Start on a small code baseKeep number of unit tests per class lowHave small classesSelect a good tool:
Configurable
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
![Page 129: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/129.jpg)
Use mutation testing from day 1 Start on a small code baseKeep number of unit tests per class lowHave small classesSelect a good tool:
ConfigurableFlexible
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
![Page 130: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/130.jpg)
Use mutation testing from day 1 Start on a small code baseKeep number of unit tests per class lowHave small classesSelect a good tool:
ConfigurableFlexibleIdentifies surviving the mutant
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
![Page 131: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/131.jpg)
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
![Page 132: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/132.jpg)
Believe the tool
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
![Page 133: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/133.jpg)
Believe the toolFix the problem
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
![Page 134: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/134.jpg)
Believe the toolFix the problemDon't turn mutation testing off
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
![Page 135: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/135.jpg)
Believe the toolFix the problemDon't turn mutation testing offLess code
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
![Page 136: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/136.jpg)
Believe the toolFix the problemDon't turn mutation testing offLess codeMore unit tests
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
![Page 137: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/137.jpg)
Believe the toolFix the problemDon't turn mutation testing offLess codeMore unit tests
More intelligent unit tests
Experiences and recommendations
Filip van Laenen
Tuesday, 22 April 14
![Page 138: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/138.jpg)
Available tools
http://en.wikipedia.org/wiki/Mutation_testing#External_links
Tuesday, 22 April 14
![Page 139: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/139.jpg)
Ruby: Heckle, Mutant
Available tools
http://en.wikipedia.org/wiki/Mutation_testing#External_links
Tuesday, 22 April 14
![Page 140: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/140.jpg)
Ruby: Heckle, MutantJava: pitest, Jumble, Jester
Available tools
http://en.wikipedia.org/wiki/Mutation_testing#External_links
Tuesday, 22 April 14
![Page 141: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/141.jpg)
Ruby: Heckle, MutantJava: pitest, Jumble, JesterC#: Nester, NinjaTurtle, Cream
Available tools
http://en.wikipedia.org/wiki/Mutation_testing#External_links
Tuesday, 22 April 14
![Page 142: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/142.jpg)
Ruby: Heckle, MutantJava: pitest, Jumble, JesterC#: Nester, NinjaTurtle, CreamPython: Pester
Available tools
http://en.wikipedia.org/wiki/Mutation_testing#External_links
Tuesday, 22 April 14
![Page 143: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/143.jpg)
Open sourceWorks with Java 5, 6, 7Works with all mocking frameworks including PowerMockIntegrates with Maven, Ant, Gradle & SBTPlugins for Eclipse & IntelliJPlugins for Jenkins & SonarQubeReleases every 3 months or so since 2011
Tuesday, 22 April 14
![Page 144: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/144.jpg)
• The Ladders - New York• Sky - Livingston• Insurance companies• Investment banks• Biotech companies• Norway's e-voting system
Tuesday, 22 April 14
![Page 145: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/145.jpg)
• The Ladders - New York• Sky - Livingston• Insurance companies• Investment banks• Biotech companies• Norway's e-voting system
Tuesday, 22 April 14
![Page 146: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/146.jpg)
• The Ladders - New York• Sky - Livingston• Insurance companies• Investment banks• Biotech companies• Norway's e-voting system
Tuesday, 22 April 14
![Page 147: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/147.jpg)
-JVM
Seb Rose,
Tuesday, 22 April 14
![Page 148: Better code through making bugs](https://reader033.vdocument.in/reader033/viewer/2022052822/554dd62cb4c905c70e8b4bba/html5/thumbnails/148.jpg)
-JVM
Seb Rose,
Availa
ble 20
14
(hopef
ully)
Tuesday, 22 April 14