kill the mutants - a better way to test your tests

KILL THE MUTANTSa better way to test your tests

ABOUT ME

• Roy van Rijn

• Mutants:

• Nora

• Lucas

• Works for

SHOW OF HANDSlet's do a

WHO DOES• Unit testing

• Test-driven development (TDD)

• Continuous integration

• Measure code coverage

• Mutation testing

UNIT TESTING• Prove your code works

• Instant regression tests

• Improve code design

• Has become a mainstream practise over the last 10 years

CONTINUOUS INTEGRATION• Automate testing

• Maintain a single source repository

• Collect statistics

CODE COVERAGE• Measure the lines (or branches) that are executed during testing

CODE COVERAGE• How did they test your car?

CODE COVERAGE• Who has seen (or written?) tests

• without verifications or assertions?

• just to fake and boost coverage?

• 100% branch coverage proves nothing

QUIS CUSTODIET IPSOS CUSTODES?Who watches the watchmen?

MUTATION TESTING• Proposed by Richard J. Lipton in 1971 (winner of 2014 Knuth Prize)

• A better way to measure the quality of your tests

• Surge of interest in the 1980s

• Time to revive this interest!

TERMINOLOGY: MUTATION• A mutation is a (small) change in your codebase, for example:

TERMINOLOGY: MUTANT

• A mutant is a mutated version of your class

MUTATION TESTING• Generate (a lot of) mutants of your codebase

• Run (some of) your unit tests

• Check the outcome!

OUTCOME #1: KILLED

• A mutant is killed if a test fails (detecting the mutated code)

• This proves the mutated code is properly tested

OUTCOME #2: LIVED

• A mutant didn’t trigger a failing test…

OUTCOME #3: TIMED OUT

• The mutant caused the program loop, get stuck

OTHER OUTCOMES• NON-VIABLE

• JVM could not load the mutant bytecode

• MEMORY ERROR

• JVM ran out of memory during test

• RUN ERROR

• An error but none of the above.

FAULT INJECTION?• With fault injection you test code

• Inject faults/mutations and see how the system reacts

• With mutation testing you test your tests

• Inject faults/mutations and see how the tests react

TOOLING• µJava: http://cs.gmu.edu/~offutt/mujava/ (inactive)

• Jester : http://jester.sourceforge.net/ (inactive)

• Jumble: http://jumble.sourceforge.net/ (inactive)

• javaLanche: http://www.st.cs.uni-saarland.de/mutation/ (inactive)

• PIT: http://pitest.org/

USING PIT

• PIT uses configurable ‘mutators'

• ASM (bytecode manipulation) is used to mutate your code

• No mutated code is stored, it can't interfere with your code

• Generates reports with test results

MUTATORS: CONDITION BOUNDARY

> into >=< into <=>= into ><= into <

MUTATORS: NEGATE CONDITIONALS

== into != != into == <= into > >= into < < into >= > into <=

MUTATORS: REMOVE CONDITIONALS

intoif(true) {

//something }

if(a == b) { //something

}

MUTATORS: MATH

+ into - - into + * into / / into * % into * & into |

<< into >> >> into <<>>> into <<<a++ into a-- a-- into a++

MUTATORS: MANY MORE

• Replacing return values (return a; becomes return 0;)

• Removal of void invocations (doSomething(); is removed)

• Some enabled by default, others are optional/configurable

MUTATION TESTING IS SLOW?

• Speed was unacceptable in the 80's

• Mutation testing is still CPU intensive

• But PIT has a lot of methods to speed it up!

WHICH TESTS TO RUN?

• PIT uses code coverage to decide which tests to run:

• A mutation is on a line covered by 3 tests? Only run those.

SIMPLE EXAMPLE

• 100 classes

• 10 unit tests per class

• 2 ms per unit test

• Total time (all tests): 100 x 10 x 2ms = 2s

SIMPLE EXAMPLE

• Total time (all tests): 100 x 10 x 2ms = 2s

• 8 mutants per class, 100 classes x 8 = 800 mutants

• Brute force: 800 x 2s = 26m40s

• Smart testing: 800 x 10 x 2ms = 16s

LONGER EXAMPLE

• Total time (all tests): 1000 x 10 x 2ms = 20s

• 8 mutants per class, 1000 classes x 8 = 8000 mutants

• Brute force: 8000 x 20s = 1d20h26m40s…!!!

• Smart testing: 8000 x 10 x 2ms = 2m40s

PERFORMANCE TIPS

• Write fast tests

• Good separation or concerns

• Use small classes, keep amount of unit tests per class low

INCREMENTAL ANALYSIS

• Experimental feature

• Incremental analysis keeps track of:

• Changes in the codebase

• Previous results

HOW ABOUT MOCKING?

• PIT has support for :

• Mockito, EasyMock, JMock, PowerMock and JMockit

HOW TO USE PIT?

• Standalone Java process

• Build: Ant task, Maven plugin

• CI: Sonarqube plugin, Gradle plugin

• IDE: Eclipse plugin (Pitclipse), IntelliJ Plugin

STANDALONE JAVA

java -cp <your classpath including pit jar and dependencies> org.pitest.mutationtest.commandline.MutationCoverageReport --reportDir /somePath/ --targetClasses com.your.package.tobemutated* --targetTests com.your.package.* --sourceDirs /sourcePath/

MAVEN PLUGIN

Run as: mvn clean package org.pitest:pitest-maven:mutationCoverage

<plugin><groupId>org.pitest</groupId><artifactId>pitest-maven</artifactId><version>1.0.0</version><configuration><targetClasses><param>com.your.package.tobemutated*</param>

</targetClasses><jvmArgs>…</jvmArgs>

</configuration></plugin>

EXAMPLELet’s kill some mutants… or be killed.

USE CASE

The price of an item is 17 euro

If you buy 20 or more, all items cost 15 euro

If you have a coupon, all items cost 15 euro

CODE

public int getPrice(int amountOfThings, boolean coupon) {if (amountOfThings >= 20 || coupon) {return amountOfThings * 15;

}return amountOfThings * 17;

}

TEST #1

@Testpublic void testNormalPricing() {//Not enough for discount:int amount = 1;Assert.assertEquals(17, businessLogic.getPrice(amount, false));

}

BRANCH COVERAGE

public int getPrice(int amountOfThings, boolean coupon) {if (amountOfThings >= 20 || coupon) {return amountOfThings * 15;

}return amountOfThings * 17;

}

TEST #2

@Testpublic void testDiscountPricingByAmount() {//Enough for discount:int amount = 100;Assert.assertEquals(1500, businessLogic.getPrice(amount, false));

}

BRANCH COVERAGE

public int getPrice(int amountOfThings, boolean coupon) {if (amountOfThings >= 20 || coupon) {return amountOfThings * 15;

}return amountOfThings * 17;

}

TEST #3

@Testpublic void testDiscountWithCoupon() {//Not enough for discount, but coupon:int amount = 1;Assert.assertEquals(15, businessLogic.getPrice(amount, true));

}

BRANCH COVERAGE

public int getPrice(int amountOfThings, boolean coupon) {if (amountOfThings >= 20 || coupon) {return amountOfThings * 15;

}return amountOfThings * 17;

}

PIT RESULT

PIT RESULT

> org.pitest.mutationtest…ConditionalsBoundaryMutator>> Generated 1 Killed 0 (0%)> KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0

PIT tells us: Changing >= into > doesn’t trigger a failing test

TEST #4

@Testpublic void testDiscountAmountCornerCase() {//Just enough for discount, mutation into > should fail this testint amount = 20;Assert.assertEquals(300, businessLogic.getPrice(amount, true));

}

BRANCH COVERAGE

public int getPrice(int amountOfThings, boolean coupon) {if (amountOfThings >= 20 || coupon) {return amountOfThings * 15;

}return amountOfThings * 17;

}

PIT RESULT

PIT RESULT

> org.pitest.mutationtest…ConditionalsBoundaryMutator>> Generated 1 Killed 0 (0%)> KILLED 0 SURVIVED 1 TIMED_OUT 0 NON_VIABLE 0 > MEMORY_ERROR 0 NOT_STARTED 0 STARTED 0 RUN_ERROR 0 > NO_COVERAGE 0

STILL WRONG!?

DID YOU SPOT THE BUG?

@Testpublic void testDiscountAmountCornerCase() {//Just enough for discount, mutation into > should fail this testint amount = 20;Assert.assertEquals(300, businessLogic.getPrice(amount, true));

}

SUMMARY

• Mutation testing automatically tests your tests

• Mutation testing can find bugs in your tests

• Code coverage is wrong, gives a false sense of security

• Mutation testing with PIT is easy to implement

QUESTIONS?