metamorphic testing - university college londoncrest.cs.ucl.ac.uk/cow/20/slides/cow20_chen.pdf ·...
TRANSCRIPT
![Page 1: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/1.jpg)
1
Metamorphic Testing
Tsong Yueh Chen Swinburne University of Technology
20th CREST Open Workshop
![Page 2: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/2.jpg)
2
Oracle Problem
• Oracle - a mechanism to verify the correctness of the outputs
• Oracle Problem – absence of oracle or too expensive to apply the oracle
![Page 3: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/3.jpg)
3
A Motivating Example
• sin function – sin(0o )=0 – sin(30o)=0.5
• Suppose the program returns: sin(29.8o )=0.51234 incorrect sin(29.8o )=0.49876 correct?
![Page 4: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/4.jpg)
4
A Motivating Example
• sin function has the following properties – sin(x)= sin(x+360) – ……
• Execute the program with sin(389.8o ) compare sin(29.8o ) and sin(389.8o )
![Page 5: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/5.jpg)
5
Metamorphic Testing (A Simplified Form)
• Source (original) test cases – generated according to certain strategies
• Follow-up test cases could be constructed from the successful test cases with reference to certain properties of the problem
• Such properties are known as metamorphic relations
![Page 6: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/6.jpg)
6
Metamorphic Testing
• Some Characteristics – Necessary properties of the problem – not
restricted to identity relations and numeric relations
– Multiple executions – at least one source test case and one follow-up test case
– Follow-up test cases may depend on the outputs of the source test cases
![Page 7: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/7.jpg)
7
Metamorphic Testing
• test case – metamorphic test group • test outcome of pass or failure – test
outcome of satisfaction or violation of a metamorphic relation
![Page 8: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/8.jpg)
8
Metamorphic Testing
• Aimed at alleviating the oracle problem • A property-based testing method –
providing a new perspective to design test cases
• Metamorphic relation
![Page 9: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/9.jpg)
9
Categories of Research in MT
• Use of MT to test application domains which have oracle problem
• Integration of MT with other software analysis and testing methods
• Its own theory
![Page 10: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/10.jpg)
10
Testing Software with Oracle Problem
• Bioinformatics programs • Embedding systems • Machine learning software • Optimization systems • …..
![Page 11: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/11.jpg)
11
Integration with Other Methods
• Slicing – metamorphic slice • Spectrum Based Fault Localizations (SBFL) • Symbolic execution – semi-proving which
supports debugging, testing and proving
![Page 12: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/12.jpg)
12
Metamorphic Slices
• Slicing may be the most important concept in software analysis
• Conventional slices are data-based • Metamorphic slices are also property-based • Many applications of slicing assume the
existence of a test oracle
![Page 13: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/13.jpg)
13
Existing SBFL Techniques
• A test suite – the test outcome of each test case is known (oracle assumed)
• A program spectrum – execution slice • A risk evaluation formula to assign a risk
value of being faulty to each statement
![Page 14: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/14.jpg)
14
SBFL Without Oracle
• slice – metamorphic slice • test case – metamorphic test group • pass or failure of a test case – satisfaction or
violation of a metamorphic test group
![Page 15: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/15.jpg)
15
Integration with Symbol Execution
• Proving metamorphic relations – metamorphic proving
• Sometimes able to prove program • Providing useful information to debug
![Page 16: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/16.jpg)
16
Theory for Metamorphic Testing
• Metamorphic Relations – Necessary properties involving multiple inputs
• Identification or generation of MRs
– Many MRs • Set of MRs treated as input domain – selection
strategies for MRs • Prioritization of MRs
![Page 17: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/17.jpg)
17
Generation of MRs
• Is it feasible to identify or generate MRs?
![Page 18: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/18.jpg)
18
Generation of MRs
• Should target at – Development of guidelines or systematic
methods for a specific type of application domains
– Development of semi-automated methods
![Page 19: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/19.jpg)
19
Prioritization of MRs
Consider sin(x)
MR1: sin(x) = sin(x + 2 π) MR2: sin(x) = -sin(x + π) MR3: sin(-x)= -sin(x) MR4: sin(x) = sin(π-x) MR5: sin(x) = -sin(2 π - x) …
![Page 20: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/20.jpg)
20
Priorization Approaches
• Usage profile • Algorithm
![Page 21: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/21.jpg)
21
Usage Profile
§ sin(x) § Electrical engineers
§ sin(x) = sin(x + 2 π)
§ Surveyors § sin(-x)= -sin(x) § sin(x) = sin(π-x)
![Page 22: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/22.jpg)
22
Algorithm § A problem may be solved by more than one
algorithm – sorting, adaptive random testing § The same algorithm may be implemented in
different ways
![Page 23: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/23.jpg)
23
Example
§ Shortest Path problem: SP(G, a, b) using forward expansion • |SP(G, a, b)| = |SP(G, a, c)| +|SP(G, c, b)| • |SP(G, a, b)| = |SP(G, b, a)|
![Page 24: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/24.jpg)
24
A Test Case Selection Strategy
• Observation: MT reveals bugs in some software that has been used and tested by conventional testing methods for a long time.
schedule and print_token • A testing method for end-users
![Page 25: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/25.jpg)
25
End-User Software Engineering § Source test case selection strategy – any
available; otherwise special values, random or ad hoc selection
§ Selection of MRs – § usage profile § end-user’s domain knowledge § coding
![Page 26: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/26.jpg)
26
Specifications
Is MT a Black-Box Method?
![Page 27: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/27.jpg)
27
Example
−+−=!5!3
)sin(53 xxxx
![Page 28: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/28.jpg)
28
Example
• MR1 sin(-x) = -sin(x) • MR2: sin(x) = sin(x + 2 π)
![Page 29: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/29.jpg)
29
Conclusion
![Page 30: Metamorphic Testing - University College Londoncrest.cs.ucl.ac.uk/cow/20/slides/COW20_Chen.pdf · Metamorphic Testing • Some Characteristics – Necessary properties of the problem](https://reader036.vdocument.in/reader036/viewer/2022070919/5fb87ee8c0105f0e57076668/html5/thumbnails/30.jpg)
30
Thanks