utility of test coverage metrics in tdd
TRANSCRIPT
Learning Objectives
• TDD Overview• Common Test Coverage Metrics• Significance of Test Coverage Metrics in TDD• Test Coverage Measurement Tools
TDD
• Software development technique in which tests are written prior to code
• If you can’t write test for what you are trying to code, then you should not be thinking about coding
• TDD reduces the gap between design and feedback (performance achieved by implementing the design)
Test Coverage Metrics
• Statement Coverage• Branch Coverage• Condition Coverage• MCDC• Data Flow Coverage• Path Coverage• Mutation Coverage
Mutation Coverage
• Mutation – small change in program• Apply mutations to program to obtain set of
mutants M1, M2 … Mn• Run test suite on each of mutants. Mutant is
killed if an error is detected• Mutation Test Coverage = No. of mutants killed /
Total no. of mutants• Has to be automated for any reasonable sized
program
Mutation Coverage
• May not be practical to consider all possible mutants
• Use mutation operators (rules) to generate mutants
• Value mutations – change values to reflect errors in reasoning about programs
• Decision mutations – change relational operators or parenthesisation in conditions
• Statement mutations – change whole line of code
Test Coverage Measurement Tools
• PIT (www.pitest.org)• Jumble (http://jumble.sourceforge.net/)• Clover
(https://www.atlassian.com/software/clover/overview)
• Cobertura (http://cobertura.github.io/cobertura/)
• EMMA (http://emma.sourceforge.net/)
PIT
• Measures line coverage and mutation coverage• Bytecode based mutant generation• Test selection is automatic – based on line
coverage• Can be integrated with Ant or Maven based
build systems• Compatible with most mocking frameworks,
except Powermock and Jmockit
Observations
• TDD does not imply that the tests are perfect• Test Coverage measurements are relevant to
TDD• Mutation coverage tells whether something is
really tested• Tools are available which make it practical to
measure mutation coverage
Future Work
• Collect more data on test coverage in TDD• Effectiveness of mutation coverage vs. other
coverage metrics• Comparison between various coverage
measurement tools• Study effect of mocking frameworks on code
coverage