using jml runtime assertion checking to automate metamorphic testing in applications without test...
Post on 21-Dec-2015
221 views
TRANSCRIPT
Using JML Runtime Assertion Checking to Automate Metamorphic Testing in
Applications without Test Oracles
Christian Murphy, Kuang Shen, Gail Kaiser
Columbia University
Chris Murphy – Columbia University 2
Problem Statement Some applications (e.g. machine learning,
simulation) do not have test oracles that indicate whether the output is correct for arbitrary input
Oracles may exist for a limited subset of the input domain, and gross errors (e.g. crashes) can be detected with certain inputs or other techniques
However, it is difficult to detect subtle (computational) errors for arbitrary inputs
Chris Murphy – Columbia University 3
Observation Even when there is no oracle in the general
case (i.e. knowing the relationship between a particular input and its output), it may be possible to know relationships between sets of inputs and their corresponding outputs
One such approach that takes advantage of this is “Metamorphic Testing” [Chen ’98]
Chris Murphy – Columbia University 4
Metamorphic Testing An approach for creating follow-up test cases
based on those that have not revealed any defects
If input x produces output f(x), then the function’s “metamorphic properties” are used to guide a transformation function t, which is applied to produce a new test case, t(x)
We can then predict the expected value of f(t(x)) based on the value of f(x) obtained from the actual execution
Chris Murphy – Columbia University 5
Metamorphic Testing Example Consider a function std_dev(A[ ])
If we permute the elements in A, the result should stay the same
If we multiply each element in A by 2, the result should be doubled
These properties can be used to create a “pseudo-oracle” for the function
Chris Murphy – Columbia University 6
Metamorphic Testing without an Oracle When a test oracle exists, we can know
whether f(t(x)) is correctBecause we have an oracle for f(x)So if f(t(x)) is as expected, then it is correct
When there is no test oracle, f(x) acts as a “pseudo-oracle” for f(t(x)) If f(t(x)) is as expected, it is not necessarily
correctHowever, if f(t(x)) is not as expected, either f(x) or f(t(x)) (or both) is wrong
Chris Murphy – Columbia University 7
Our Solution To address the problem of applications that have
no test oracle, we use Metamorphic Testing at the function level
To specify the metamorphic properties, we use extensions to the Java Modeling Language (JML) and a pre-processor called Corduroy
To automate the process and address the need for initial input values, we check the properties at runtime
Chris Murphy – Columbia University 8
Related Work Applying Metamorphic Testing to “non-testable
programs” (applications without test oracles) Chen IST’02, Chen ISSTA’02, Chan JWSR’07
Checking algebraic specifications at runtime Sankar ISSTA’91, Nunes ICFEM’06
Checking program invariants at runtime Flanagan FME’01, Hangal ICSE’02
Chris Murphy – Columbia University 9
JML Basics Behavioral interface specification language that
embraces the Design by Contract approach
Developers can specify: preconditions (“requires”) postconditions (“ensures”) arbitrary assertions
Specifications can be checked dynamically using JML Runtime Assertion Checking tools
Chris Murphy – Columbia University 10
ProcessSpecify functions’ metamorphic
properties using extension to JML
Specify functions’ metamorphicproperties using extension to JML
Methods that checkmetamorphic properties
Methods that checkmetamorphic properties
Pure JML specificationsthat invoke test methods
Pure JML specificationsthat invoke test methods
CorduroyCorduroy
Chris Murphy – Columbia University 11
Metamorphic Properties in JML
/*@ @ensures \result <= 1 && \result >= -1; @meta sine(x + 2 * PI) == \result @meta -1 * sine(-x) == \result */public double sine (double x) { ...}
Chris Murphy – Columbia University 12
Extension to JML: Array Functions/*@ @ensures \result >= 0; @meta std_dev(\shuffle(A)) ==
\result @meta std_dev(\multiply(A, 2)) ==
\result * 2 */public double std_dev(double[] A) { ...}
Chris Murphy – Columbia University 13
Extensions to JML Array/Collection Functions
Shuffle/permute the order of the elementsMultiply each element by a constantAdd a constant to each element Include a new elementExclude an element
Addressing non-determinismCheck in range of possible valuesCheck in a set of possible values
Chris Murphy – Columbia University 14
Model
CallerFunction
CallerFunction
Input(s)Input(s)
CalleeFunction
CalleeFunction
OutputOutput
CheckMetamorphic
Properties
CheckMetamorphic
Properties
Output
Input(s)
Chris Murphy – Columbia University 15
Implementation/*@ @meta average(\multiply(A, 2)) == \result * 2 */public double average(double[] A) { ...}
protected boolean __metaTest_average (double[] A, double result) { return JMLDouble.approximatelyEqualTo (average(Corduroy.multiply(A, 2)), result * 2);}
@ensures __metaTest_average(A, \result);
Chris Murphy – Columbia University 16
Case Studies We investigated the WEKA and RapidMiner
toolkits for Machine Learning in Java
For WEKA, we tested four apps:Naïve Bayes, Support Vector Machines (SVM),
C4.5 Decision Tree, and k-Nearest Neighbors For RapidMiner, we tested one app:
Naïve Bayes
Chris Murphy – Columbia University 17
Experimental Setup For each of the five apps, we specified 4-6
metamorphic properties of selected methods (based on our knowledge of the expected behavior of the overall application)
Testing was conducted using data sets from UCI Machine Learning Repository
Goal was to determine whether the properties held as expected
Chris Murphy – Columbia University 18
Findings Discovered defects in WEKA k-NN and
WEKA Naïve Bayes related to modifying the machine learning “model”This was the result of a variable not being
updated appropriately
Discovered a defect in RapidMiner Naïve Bayes related to determining confidence There was an error in the calculation
Chris Murphy – Columbia University 19
Limitations Specifying and checking the properties was
simplified by use of the tool, but still requires some knowledge of the app to determine the properties in the first place
The approach works well for functions that take input and produce output, but in some case requires more complex properties than can be expressed in JML
Chris Murphy – Columbia University 20
Future Work Reducing testing time by checking
properties in parallel
Implementations for other languages
More empirical studies to determine effectiveness and applicability to other domains
Chris Murphy – Columbia University 21
Summary We have presented a testing approach that uses
metamorphic testing of individual functions in applications that do not have test oracles
These are specified via an extension to the JML specification language
We have also presented an implementation framework called Corduroy, which converts the specification of metamorphic properties into test methods
Using JML Runtime Assertion Checking to Automate Metamorphic Testing in
Applications without Test Oracles
Chris Murphy
Chris Murphy – Columbia University 23
Categories of Metamorphic Properties Additive: Increase (or decrease) numerical values
by a constant Multiplicative: Multiply numerical values by a
constant Permutative: Randomly permute the order of
elements in a set Invertive: Reverse the order of elements in a set Inclusive: Add a new element to a set Exclusive: Remove an element from a set
ML apps such as ranking, classification, and anomaly detection exhibit these properties [Murphy SEKE’08]
Chris Murphy – Columbia University 24
Specifying More Complex Properties/*@ @meta test_myFunc(x, \result)*/public double myFunc (int x) { ...}
private boolean test_myFunc (int x, double result){ ...}
Chris Murphy – Columbia University 25
Addressing Side Effects// holds result of last call to “average”private double value;
/*@ @assignable value; @meta average(\multiply(A, 2)) == value * 2 */public void average(double[] A) { ... value = ... // no return value}
Chris Murphy – Columbia University 26
Addressing Side Effectsprotected boolean __metaTest_average (double[] A) { double __value = value; // backup try { return JMLDouble.approximatelyEqualTo (average(Corduroy.multiply(A, 2)), value * 2); } finally { value = __value; // restore }}
__value