![Page 1: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/1.jpg)
KILL THE MUTANTSa better way to test your tests
![Page 2: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/2.jpg)
ABOUT ME
• Roy van Rijn
• Mutants:
• Nora
• Lucas
• Works for
![Page 3: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/3.jpg)
SHOW OF HANDSlet's do a
![Page 4: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/4.jpg)
WHO DOES• Unit testing
• Test-driven development (TDD)
• Continuous integration
• Measure code coverage
• Mutation testing
![Page 5: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/5.jpg)
UNIT TESTING• Prove your code works
• Instant regression tests
• Improve code design
• Has become a mainstream practise over the last 10 years
![Page 6: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/6.jpg)
CONTINUOUS INTEGRATION• Automate testing
• Maintain a single source repository
• Collect statistics
![Page 7: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/7.jpg)
CODE COVERAGE• Measure the lines (or branches) that are executed during testing
![Page 8: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/8.jpg)
CODE COVERAGE• How did they test your car?
![Page 9: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/9.jpg)
CODE COVERAGE• Who has seen (or written?) tests
• without verifications or assertions?
• just to fake and boost coverage?
• 100% branch coverage proves nothing
![Page 10: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/10.jpg)
QUIS CUSTODIET IPSOS CUSTODES?Who watches the watchmen?
![Page 11: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/11.jpg)
MUTATION TESTING• Proposed by Richard J. Lipton in 1971 (winner of 2014 Knuth Prize)
• A better way to measure the quality of your tests
• Surge of interest in the 1980s
• Time to revive this interest!
![Page 12: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/12.jpg)
TERMINOLOGY: MUTATION• A mutation is a (small) change in your codebase, for example:
![Page 13: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/13.jpg)
TERMINOLOGY: MUTANT
• A mutant is a mutated version of your class
![Page 14: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/14.jpg)
MUTATION TESTING• Generate (a lot of) mutants of your codebase
• Run (some of) your unit tests
• Check the outcome!
![Page 15: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/15.jpg)
OUTCOME #1: KILLED
• A mutant is killed if a test fails (detecting the mutated code)
• This proves the mutated code is properly tested
![Page 16: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/16.jpg)
OUTCOME #2: LIVED
• A mutant didn’t trigger a failing test…
![Page 17: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/17.jpg)
OUTCOME #3: TIMED OUT
• The mutant caused the program loop, get stuck
![Page 18: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/18.jpg)
OTHER OUTCOMES• NON-VIABLE
• JVM could not load the mutant bytecode
• MEMORY ERROR
• JVM ran out of memory during test
• RUN ERROR
• An error but none of the above.
![Page 19: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/19.jpg)
FAULT INJECTION?• With fault injection you test code
• Inject faults/mutations and see how the system reacts
• With mutation testing you test your tests
• Inject faults/mutations and see how the tests react
![Page 20: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/20.jpg)
TOOLING• µJava: http://cs.gmu.edu/~offutt/mujava/ (inactive)
• Jester : http://jester.sourceforge.net/ (inactive)
• Jumble: http://jumble.sourceforge.net/ (inactive)
• javaLanche: http://www.st.cs.uni-saarland.de/mutation/ (inactive)
• PIT: http://pitest.org/
![Page 21: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/21.jpg)
USING PIT
• PIT uses configurable ‘mutators'
• ASM (bytecode manipulation) is used to mutate your code
• No mutated code is stored, it can't interfere with your code
• Generates reports with test results
![Page 22: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/22.jpg)
MUTATORS: CONDITION BOUNDARY
> into >=< into <=>= into ><= into <
![Page 23: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/23.jpg)
MUTATORS: NEGATE CONDITIONALS
== into != != into == <= into > >= into < < into >= > into <=
![Page 24: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/24.jpg)
MUTATORS: REMOVE CONDITIONALS
intoif(true) {
//something }
if(a == b) { //something
}
![Page 25: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/25.jpg)
MUTATORS: MATH
+ into - - into + * into / / into * % into * & into |
<< into >> >> into <<>>> into <<<a++ into a-- a-- into a++
![Page 26: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/26.jpg)
MUTATORS: MANY MORE
• Replacing return values (return a; becomes return 0;)
• Removal of void invocations (doSomething(); is removed)
• Some enabled by default, others are optional/configurable
![Page 27: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/27.jpg)
MUTATION TESTING IS SLOW?
• Speed was unacceptable in the 80's
• Mutation testing is still CPU intensive
• But PIT has a lot of methods to speed it up!
![Page 28: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/28.jpg)
WHICH TESTS TO RUN?
• PIT uses code coverage to decide which tests to run:
• A mutation is on a line covered by 3 tests? Only run those.
![Page 29: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/29.jpg)
SIMPLE EXAMPLE
• 100 classes
• 10 unit tests per class
• 2 ms per unit test
• Total time (all tests): 100 x 10 x 2ms = 2s
![Page 30: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/30.jpg)
SIMPLE EXAMPLE
• Total time (all tests): 100 x 10 x 2ms = 2s
• 8 mutants per class, 100 classes x 8 = 800 mutants
• Brute force: 800 x 2s = 26m40s
• Smart testing: 800 x 10 x 2ms = 16s
![Page 31: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/31.jpg)
LONGER EXAMPLE
• Total time (all tests): 1000 x 10 x 2ms = 20s
• 8 mutants per class, 1000 classes x 8 = 8000 mutants
• Brute force: 8000 x 20s = 1d20h26m40s…!!!
• Smart testing: 8000 x 10 x 2ms = 2m40s
![Page 32: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/32.jpg)
PERFORMANCE TIPS
• Write fast tests
• Good separation or concerns
• Use small classes, keep amount of unit tests per class low
![Page 33: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/33.jpg)
INCREMENTAL ANALYSIS
• Experimental feature
• Incremental analysis keeps track of:
• Changes in the codebase
• Previous results
![Page 34: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/34.jpg)
HOW ABOUT MOCKING?
• PIT has support for :
• Mockito, EasyMock, JMock, PowerMock and JMockit
![Page 35: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/35.jpg)
HOW TO USE PIT?
• Standalone Java process
• Build: Ant task, Maven plugin
• CI: Sonarqube plugin, Gradle plugin
• IDE: Eclipse plugin (Pitclipse), IntelliJ Plugin
![Page 36: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/36.jpg)
STANDALONE JAVA
java -cp <your classpath including pit jar and dependencies> org.pitest.mutationtest.commandline.MutationCoverageReport --reportDir /somePath/ --targetClasses com.your.package.tobemutated* --targetTests com.your.package.* --sourceDirs /sourcePath/
![Page 37: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/37.jpg)
MAVEN PLUGIN
Run as: mvn clean package org.pitest:pitest-maven:mutationCoverage
<plugin><groupId>org.pitest</groupId><artifactId>pitest-maven</artifactId><version>1.0.0</version><configuration><targetClasses><param>com.your.package.tobemutated*</param>
</targetClasses><jvmArgs>…</jvmArgs>
</configuration></plugin>
![Page 38: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/38.jpg)
EXAMPLELet’s kill some mutants… or be killed.
![Page 39: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/39.jpg)
USE CASE
The price of an item is 17 euro
If you buy 20 or more, all items cost 15 euro
If you have a coupon, all items cost 15 euro
![Page 40: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/40.jpg)
CODE
public int getPrice(int amountOfThings, boolean coupon) {if (amountOfThings >= 20 || coupon) {return amountOfThings * 15;
}return amountOfThings * 17;
}
![Page 41: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/41.jpg)
TEST #1
@Testpublic void testNormalPricing() {//Not enough for discount:int amount = 1;Assert.assertEquals(17, businessLogic.getPrice(amount, false));
}
![Page 42: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/42.jpg)
BRANCH COVERAGE
public int getPrice(int amountOfThings, boolean coupon) {if (amountOfThings >= 20 || coupon) {return amountOfThings * 15;
}return amountOfThings * 17;
}
![Page 43: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/43.jpg)
TEST #2
@Testpublic void testDiscountPricingByAmount() {//Enough for discount:int amount = 100;Assert.assertEquals(1500, businessLogic.getPrice(amount, false));
}
![Page 44: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/44.jpg)
BRANCH COVERAGE
public int getPrice(int amountOfThings, boolean coupon) {if (amountOfThings >= 20 || coupon) {return amountOfThings * 15;
}return amountOfThings * 17;
}
![Page 45: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/45.jpg)
TEST #3
@Testpublic void testDiscountWithCoupon() {//Not enough for discount, but coupon:int amount = 1;Assert.assertEquals(15, businessLogic.getPrice(amount, true));
}
![Page 46: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/46.jpg)
BRANCH COVERAGE
public int getPrice(int amountOfThings, boolean coupon) {if (amountOfThings >= 20 || coupon) {return amountOfThings * 15;
}return amountOfThings * 17;
}
![Page 47: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/47.jpg)
PIT RESULT
![Page 48: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/48.jpg)
PIT RESULT
> org.pitest.mutationtest…ConditionalsBoundaryMutator>> Generated 1 Killed 0 (0%)> KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0
PIT tells us: Changing >= into > doesn’t trigger a failing test
![Page 49: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/49.jpg)
TEST #4
@Testpublic void testDiscountAmountCornerCase() {//Just enough for discount, mutation into > should fail this testint amount = 20;Assert.assertEquals(300, businessLogic.getPrice(amount, true));
}
![Page 50: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/50.jpg)
BRANCH COVERAGE
public int getPrice(int amountOfThings, boolean coupon) {if (amountOfThings >= 20 || coupon) {return amountOfThings * 15;
}return amountOfThings * 17;
}
![Page 51: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/51.jpg)
PIT RESULT
![Page 52: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/52.jpg)
PIT RESULT
> org.pitest.mutationtest…ConditionalsBoundaryMutator>> Generated 1 Killed 0 (0%)> KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0
STILL WRONG!?
![Page 53: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/53.jpg)
DID YOU SPOT THE BUG?
@Testpublic void testDiscountAmountCornerCase() {//Just enough for discount, mutation into > should fail this testint amount = 20;Assert.assertEquals(300, businessLogic.getPrice(amount, true));
}
![Page 54: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/54.jpg)
SUMMARY
• Mutation testing automatically tests your tests
• Mutation testing can find bugs in your tests
• Code coverage is wrong, gives a false sense of security
• Mutation testing with PIT is easy to implement
![Page 55: Kill the mutants - A better way to test your tests](https://reader030.vdocument.in/reader030/viewer/2022013004/55d57950bb61eb9f2f8b4646/html5/thumbnails/55.jpg)
QUESTIONS?