test oracles and test script generation in combinatorial ...iwct2016.unibg.it/slides/kruse.pdftest...
TRANSCRIPT
![Page 1: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/1.jpg)
Test Oracles and Test Script Generation in Combinatorial Testing
Peter M. Kruse
Berner & Mattner Systemtechnik GmbH
Berlin, Germany
![Page 2: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/2.jpg)
Overview
• Classification Tree Method
• Expected results
• Executable test scripts
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 2
![Page 3: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/3.jpg)
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 3
Combinatorial Testing Background
• Combinatorial Interaction Testing (CIT) is a black box system testing technique that samples inputs, configurations and parameters and combines them in a systematic fashion.
• Creating functional tests derived from software's specifications Thomas J. Ostrand and Marc J. Balcer. The Category-Partition Method for specifying and generating functional tests, 1988
• Coverage Criterion: Minimum, Maximum, Pairwise, N-Wise
• Pairwise NP Complete Yu Lei, Kuo-Chung Tai. In-parameter-order: a test generation strategy for pairwise testing, 1998
• N-wise NP Complete Alan W. Williams and Robert L. Probert. A measure for component interaction test coverage, 2001
• Constraints
Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. Interaction testing of highly-configurable systems in the presence of constraints, 2007
![Page 4: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/4.jpg)
Classification Tree Method
• Grochtmann/Grimm 1993, Daimler Research
Two Steps:
• 1st Design Classification Tree – One Classification per test aspect
(Parameter)
– One Class for each parameter value
– Resulting in a Tree of Classifications
• 2nd Compose Test Cases – Can be automated using TESTONA tool
(formerly Classification Tree Editor)
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 4
![Page 5: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/5.jpg)
Example
• Test object: Database Management System
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 5
Select a test object: decompose
5
![Page 6: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/6.jpg)
Input domain
Determine input data space
Select a test object: decompose
TESTONA – Test Design using the Classification Tree Method • October 2015
Example
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 6
![Page 7: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/7.jpg)
Privileges Operation Access Method
Input domain
Identify relevant aspects (e.g. from specification)
Determine input data space
Select a test object: decompose
TESTONA – Test Design using the Classification Tree Method • October 2015
Example
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 7
![Page 8: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/8.jpg)
Input domain Privileges Operation Access Method
Classify the input data space into classes
TESTONA – Test Design using the Classification Tree Method • October 2015
Example
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 8
![Page 9: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/9.jpg)
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 9
1 2 3
Input domain Privileges Operation Access Method
Classify the input data space into classes
Combine classes into test cases
TESTONA – Test Design using the Classification Tree Method • October 2015
Example
![Page 10: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/10.jpg)
Test Oracle
• Howden (1978) non-trivial challenge of deciding whether a test case has passed or failed
• Categorization by Barr, Harman, McMinn, Shahbaz, and Yoo (2015) – non-automated
– implicit
– derived
– specified
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 10
![Page 11: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/11.jpg)
11 2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing
Non-automated Oracles
• Trivial case
• Mapping function f(tq) = Rq unknown
Implicit Oracles
• Indirect evaluation (e.g. no exception)
• Mapping function
Rq = R1 = R2 = … = Rn
Derived Oracles
• A posteriori (e.g. Regression Test, back-to-back Test)
• Mapping function f(tq) = Rq not needed
Specified Oracles
• Based on (formal) Specification
• Mapping function f(tq) = Rq known
![Page 12: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/12.jpg)
Non-automated Oracle Example
• Manual assignment for each test case
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 12
![Page 13: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/13.jpg)
Test Oracles and Test Script Generation in Combinatorial Testing
13
Implicit Oracle Example
2016-04-10
No exceptions at runtime, …
• Monitor system
![Page 14: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/14.jpg)
Test Oracles and Test Script Generation in Combinatorial Testing
14
Derived Oracle Example
2016-04-10
• Run once, record results, assign to tests
![Page 15: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/15.jpg)
Test Oracles and Test Script Generation in Combinatorial Testing
15
Specified Oracle Example
C2 ↔ R1
C1 ʌ ( C3 v (C4 ʌ ¬C10)) ↔ R2
2016-04-10
![Page 16: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/16.jpg)
Specified Oracle Example
Using constraints can be problematic
1. Makes computation of output more difficult, solver must be used
2. possible to write incomplete/inconsistent mappings (e.g. C1 ↔ R1 and C1 ↔ R2)
3. allows non-determinism
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 16
TESTONA has build in Solver
Task of test designer
![Page 17: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/17.jpg)
Test Oracles and Test Script Generation in Combinatorial Testing
2016-04-10
Test Script Generation
• Problem: How to execute test specification?
17
![Page 18: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/18.jpg)
2016-04-10 18 Test Oracles and Test Script Generation in
Combinatorial Testing
Test Script Generation @Test public void %TEST% throws Exception {
}
import org.junit.Assert.*; public class Example {
}
Include these per mark in test case
Include this per test case
Include this once at start
![Page 19: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/19.jpg)
2016-04-10 19 Test Oracles and Test Script Generation in
Combinatorial Testing
Test Script Generation @Test public void %TEST% throws Exception {
}
import org.junit.Assert.*; public class Example {
}
Include these per mark in test case
Include this per test case
Include this once at start
![Page 20: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/20.jpg)
Effort Considerations
• Instead of implementing each test case
• Implement each parameter
• Less effort
• Reusable
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 20
To be evaluated large scale
Formalization in the paper
![Page 21: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/21.jpg)
Related Work
• Most work on CT focuses on the calculation of minimal size test suites, mostly pairwise.
• Some CT approaches consider test oracles.
• Reports on test implementation and test execution are limited.
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 21
![Page 22: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/22.jpg)
Non-automated Oracles
• All approaches
Implicit Oracles
• AETG, ACTS
Derived Oracle
• ACTS
Specified Oracles
• AETG, PICT, ACTS, ATGT, FoCuS
Related Work – Oracles
“Oracle-free Testing“ IWCT‘15
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 22
![Page 23: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/23.jpg)
Related Work Test Implementation
• Integrated Approach also for – Category Partition Method (as part of specification
language)
– FoCuS (using post-pressing with templates)
• External Solutions – AETG (use of perl script reported)
– PICT
– ACTS
– …
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 23
![Page 24: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/24.jpg)
Future Work
• Large scale evaluation
• Effort for different test script generation approaches with focus on test suite maintenance
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 24
– costs of adapting existing test suites, test scripts, post-processors when parameters are added, modified or removed
• What about model import? – UML, Statecharts, …
![Page 25: Test Oracles and Test Script Generation in Combinatorial ...iwct2016.unibg.it/slides/kruse.pdfTest Oracles and Test Script Generation in Combinatorial Testing 3 Combinatorial Testing](https://reader030.vdocument.in/reader030/viewer/2022040211/5e6f83d0283404117822ebb2/html5/thumbnails/25.jpg)
Conclusion
• Post-processing is avoided in our approach
• Test implementation allows for the direct execution of combinatorial test suites
• Independent of different SUT types
• Several kinds of oracles are supported
2016-04-10 Test Oracles and Test Script Generation in
Combinatorial Testing 25
Peter M. Kruse Berner & Mattner Systemtechnik GmbH