mutants, tests and zombies - testcon moscow · a mutation is a controlled change in the sut which...
TRANSCRIPT
Mutants, tests and zombies
Alexander Todorovhttp://atodorov.org
@atodorov_TestCon Moscow 2018
S
Is my test suite good enough
?
What is a mutation ?
- if age > 18 then
+ if age < 18 then
buy(beer)
Федеральный закон о государственном регулировании
производства и оборота этилового спирта, алкогольной
и спиртосодержащей продукции says 18+
- had_lunch = True
+ had_lunch = False
while True do
- break
+ continue
Definition
A mutation is a controlled change in the SUT
which produces a valid syntax but changes
program behavior!
Mutation testing can be used to evaluate the
quality of existing test suites!
It can find missing or inadequate tests or dead
code!
Algorithm for mutation testing
for operator in mutation-operators:
for site in operator.sites(code):
for mutant in operator.mutate(site):
run_tests(mutant)
https://github.com/sixty-north/cosmic-ray
3 rules to kill mutants !
run_tests(vanilla_code) MUST PASS
if run_tests(mutated_code) == FAIL:
mutant dies
if run_tests(mutated_code) == PASS:
zombie
class Desk < ApplicationRecord
... skip ...
def upcase_language_code
language_code&.upcase
end
end
@@ -1,4 +1,4 @@
def upcase_language_code
- language_code&.upcase
+ nil
end
еxpect(
desk.upcase_language_code
).to_not be_nil
@@ -1,4 +1,4 @@
def upcase_language_code
- language_code&.upcase
+ self
end
еxpect(
desk.upcase_language_code
).to be_instance_of(String)
@@ -1,4 +1,4 @@
def upcase_language_code
- language_code&.upcase
+ language_code
end
desk.language_code = 'ru'
expect(
desk.upcase_language_code
).to eq('RU')
@@ -1,4 +1,4 @@
def upcase_language_code
- language_code&.upcase
+ language_code.upcase
end
desk.language_code = nil
expect(
desk.upcase_language_code
).to be_nil
Is my test suite good enough
?
http://kaner.com/pdfs/negligence_and_testing_coverage.pdf
Mutation testing vs. coverage
●http://bit.ly/GTAC2015mutation
●http://bit.ly/GTAC2016coverage
●http://bit.ly/MutationVsCoverage
SUT: pelican-ab, v0.2.1
AB_EXPERIMENT="control" make github
AB_EXPERIMENT="123" make github
AB_EXPERIMENT="xy" make github
●100% branch & mutation coverage
●Bug when
DELETE_OUTPUT_DIRECTORY=True
SUT: pelican-ab, v0.2.1
●Recursion loop bug when class is inherited
- super(self.__class__, self).....
+ super(PelicanAbExperimentWriter, self)....
●See Pylint #1109 for more info
100 % of any metric
is useless
How slow is it ?
https://github.com/rhinstaller/pykickstart/
●Full mutation test run: > 6 days
●After divide and conquer & fail fast: 6.5 hrs
When to use
mutation testing ?
Mutation testing tools
●Python – sixty-north/cosmic-ray
●Ruby – mbj/mutant
●Java – hcoles/pitest
●JavaScript - stryker-mutator/stryker
●PHP - padraic/humbug
●LLVM based: C, C++, Rust, etc – mull-
project/mull
https://mutation-testing.slack.com/
СПАСИБО