automatic!test!generation! · 2018-04-05 · goals!of!a!test!suite! all!instructions!...
TRANSCRIPT
![Page 1: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/1.jpg)
Automatic Test Generation
Galeotti/Gorla/Rau Saarland University
![Page 2: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/2.jpg)
Testing : Find inputs that make the program fail
Debugging : The process of finding the cause of a failure.
![Page 3: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/3.jpg)
Test Case Values/Test Input/Test Data : Values that satisfy a test requirement
Expected output: The result of the test case if the software under test behaves as expected. Test Oracle
![Page 4: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/4.jpg)
Goals of a Test Suite All instructions All conditions (decisions) All paths All inputs
Goals might not be feasible Example: dead code Detection of unfeasible goals is undecidable in general
100% goal coverage might not be possible
![Page 5: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/5.jpg)
To generate automatically tests
Satisfying adequacy criteria
Avoid writting the tests ourselves
Find bugs faster
Update tests automatically if the software evolves
![Page 6: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/6.jpg)
It requires: Choosing adequacy criteria Generate inputs Check output
code test generator • automated
testing oracle
0
1
3
2
0 3 2
0 3 2
0 3
pass
fail
![Page 7: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/7.jpg)
Extend the spectra of possible inputs in a test case We have no idea where the failure could be
Manual Test Cases
Randomized Testing
![Page 8: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/8.jpg)
Idea: See how an application endures “noise” in the input
Inspired in real life: Unix ssh and vi use in a dial-‐up connection.
Fuzzers Tools that create mal-‐formed random inputs to a program ▪ To crash the program ▪ To find security flaws
![Page 9: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/9.jpg)
File formats e.g. documents, pictures
Network protocols servers and clients
Specific interfaces e.g. DOM
![Page 10: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/10.jpg)
The Lexer can be exercised in detail
But few inputs reach the parser…
And even less inputs reach the runtime …
![Page 11: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/11.jpg)
Create program inputs randomly
Observe if the program behaves “correctly” Using explicit contracts (pre & posts) Implicitly: runtime undeclared exceptions
Advantages: Easy to implement Good coverage if the test suite is big enough
![Page 12: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/12.jpg)
int double (int v) {
return 2*v; } void testMe (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR; } }
}
Possible inputs: x = 2, y = 10 ▪ First IF is false
x = 102, y = -‐10 ▪ First IF is false
x = 4, y = 2 ▪ First IF is true
x = 34, y = 0 ▪ First IF is false
x = 40, y = 20 ▪ It hits ERROR
![Page 13: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/13.jpg)
Given m(x1:T1,…,xk:Tk) a method under test For each xi:
If Ti is a primitive data type ▪ A random primitive value
If Ti is a reference, choose randomly among: ▪ The null value ▪ The constructor with no arguments (if it exists)
![Page 14: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/14.jpg)
Given this method:
We generate random values for a and b: a=124, b=3 a=-‐15, b=0
!public static int max(int a, int b) {!! !if (a>b) { !! ! !return a;!! !}!! !else {!! ! !return b;!! !}!!}!
![Page 15: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/15.jpg)
Given this method:
We can use LinkedList() or null list=null list=[] (the empty list)
public static Integer largest(LinkedList<Integer> list) { int index = 0; int max = Integer.MIN_VALUE; while (index <= list.size()-‐1) { if (list.get(index) > max) { max = list.get(index); } index++; } return max; }
![Page 16: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/16.jpg)
Each test sequence is independent from the other sequences (redundancy)
Complex objects tend to be rather simple
![Page 17: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/17.jpg)
Given m(x1:T1,…,xk:Tk):Tk+1 a method under test
For each xi: If Ti is a primitive data type ▪ A random primitive value ▪ A method m’(y1:T’1,…yj:T’j): Ti (the return value is Ti) ▪ Recursively apply the same procedure to y1,…,yj
If Ti is a reference, choose randomly among: ▪ The null value ▪ A method m’(y1:T’1,…yj:T’j): Ti (the return value is Ti) ▪ Recursively apply the same procedure to y1,…,yj
![Page 18: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/18.jpg)
Program under test:
Random test sequence: int b = 254; LinkedList<Integer> l = new LinkedList<Integer>(); l.add(0,2); Integer x = l.get(0); int a = x.intValue(); max(a,b)
public static int max(int a, int b) { if (a>b) { return a; } else { return b; } }
![Page 19: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/19.jpg)
Method under test:
Random test sequence:
public static Integer largest(LinkedList<Integer> list) { int index = 0; int max = Integer.MIN_VALUE; while (index <= list.size()-‐1) { if (list.get(index) > max) { max = list.get(index); } index++; } return max; }
LinkedList<Integer> list0 = new LinkedList<Integer>(); list0.add(0,2); int int0 = list0.getFirst(); list0.add(0,int0); largest(l);
![Page 20: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/20.jpg)
How do we know a test is “interesting” We know some sort of Test Oracle
How do we select non-‐redundant tests? More generation time => More generated tests Test Selection is very expensive
![Page 21: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/21.jpg)
Randomized Test Generator for Object-‐Oriented Programs
Execution Feedback
![Page 22: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/22.jpg)
Object level: 0.equals(0) returns true 0.equals(0) throws no exception o.hashCode() throws no exception o.toString() throws no exception
Method level: Method throws no AssertionError Method throws no NullPointerException if all args !=null
User can add more contracts (Randoop interface)
![Page 23: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/23.jpg)
Randoop classifies generated tests into: Failing Test Cases: ▪ Execution led to a contract failure
Normal Test Cases: ▪ Execution did not violate any contract
All failing tests are collected.
Normal tests are filtered
![Page 24: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/24.jpg)
Equality The equals() method is used to see if a given object was created before
A pool with all created objects is kept alive. Null
Avoid using a null return value (use “null” directly instead).
Exceptions If the test leads to a exception, do not extend that.
![Page 25: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/25.jpg)
The execution of these two sequences is redundant
Smaller size improves program understanding.
Date d = new Date(2006,2,14);!d.setMonth(-1); !
Date d = new Date(2006,2,14);!d.setMonth(-1); !d.setDay(5);!
![Page 26: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/26.jpg)
The execution of these two sequences is redundant
Both hashSets are equal w.r.t. the equals() method.
HashSet s0 = new HashSet();!Integer int0 = new Integer(1);!Integer int1 = new Integer(0);!s0.add(int0);!s0.add(int1);!
HashSet s0 = new HashSet();!Integer int0 = new Integer(0);!Integer int1 = new Integer(1);!s0.add(int0);!s0.add(int1);!
![Page 27: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/27.jpg)
Test case exhibiting a failure Fails on Sun 1.5, 1.6.
public static void test1() { LinkedList l1 = new LinkedList(); Object o1 = new Object(); l1.addFirst(o1); TreeSet t1 = new TreeSet(l1); Set s1=Collections.unmodifiableSet(t1); Assert.assertTrue(s1.equals(s1)); }
![Page 28: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/28.jpg)
Regression Test Passes on Sun 1.5, fails on Sun 1.6 Beta 2.
public static void test2() { BitSet b = new BitSet(); Assert.assertEquals(64, b.size()); b.clone(); Assert.assertEquals(64, b.size()); }
![Page 29: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/29.jpg)
Advantages: Few requirements Unbiased
Disavantages: It does not benefit from source code information. It is difficult to find “deep” errors.
![Page 30: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/30.jpg)
Generate all non-‐isomorphic valid inputs up to a given size.
Use programmatic contracts to decide if an input is valid.
Prune search space efficiently.
![Page 31: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/31.jpg)
Type information The class declaration states the values a field can take
class BinaryTree { Node root; class Node { Node left; Node right; } }
Enumerate all possible values up to a given length.
![Page 32: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/32.jpg)
1. Select k the maxium input size
2. Generate all inputs up to size k
3. Discard all inputs that do not satisfy the precondition
4. Execute the program
5. Check the postcondition
![Page 33: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/33.jpg)
Binary trees How many instances of k<=3?
3 nodes 2 values per node (left, right) 4 possible values (consider also null) for each node and the tree instance.
4 * (4 * 4) ^3 = 16.384 instances! For 4 nodes it grows to more than 1.000.000!
Number of instances grow exponentially.
class BinaryTree { Node root; class Node { Node left; Node right; } }
![Page 34: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/34.jpg)
This enumeration does not take into account many important aspects:
Many inputs are not trees
Many inputs are isomorphic instances.
![Page 35: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/35.jpg)
Automated Testing Based on Java Predicates
http://korat.sourceforge.net/index.html
Efficiently enumerates instances by: Monitoring which fields are accessed ▪ Which values should I change?
Avoiding isomorphic instances ▪ Vector representation of the heap configuration.
![Page 36: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/36.jpg)
boolean repOk() {! if (this.root == null) !
!return true;! Set visited = new HashSet(); ! visited.add(t.root);! List workList = new LinkedList(); ! workList.add(t.root);! while (!workList.isEmpty()) {! Node current = (Node)workList.removeFirst();! if (current.left != null) {! if (!visited.add(current.left)) ! return false;! workList.add(current.left); }! if (current.right != null) {! if (!visited.add(current.right)) ! return false;! workList.add(current.right); }! } ! !! return true;!}!
![Page 37: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/37.jpg)
Isomorphic instances Identical except object names
root
N0 left right
N1 N2
T0
root
N2 left right
N1 N0
T0
root
N1 left right
N2 N0
T0
![Page 38: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/38.jpg)
public static Finitization finBinaryTree(int NUM_Node) { !!Finitization f = new Finitization(BinaryTree.class); !!ObjSet nodes = f.createObjects("Node", NUM_Node); !!nodes.add(null);!!f.set("root", nodes); // root in null + Node!!f.set("Node.left", nodes); // Node.left in null + Node !!f.set("Node.right", nodes); // Node.right in null+ Node!!return f;!
}!
![Page 39: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/39.jpg)
null null null null null null null
class BinaryTree { Node root; class Node { Node left; Node right; } }
this
root left right
BT0: BinaryTree
N0: Node
N1: Node
N2: Node
BinaryTree instances
Node instances
N0
left right
N1
left right
N2
![Page 40: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/40.jpg)
root
N0 left right
N1 N2
T0
root
N2 left right
N1 N0
T0
root
N1 left right
N2 N0
T0
N0 N1 N2 null null null null
root
left
right
left
right
left
right
N0 N1 N2
N1 null null N2 N0 null null
root
left
right
left
right
left
right
N0 N1 N2
N2 null null null null N1 N0 root
left
right
left
right
left
right
N0 N1 N2
![Page 41: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/41.jpg)
null null null null null null null
this
root left right
T0
N0
left right
N1
left right
N2
null
root
this.repOk()==true (move right)
this.root
![Page 42: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/42.jpg)
N0 null null null null null null
this
root left right
T0
N0
left right
N1
left right
N2
root
N0
null null
left right
this.repOk()==true (move right)
this.root
N0.left
N0.right
![Page 43: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/43.jpg)
N0 N0 null null null null null
this
root left right
N0
left right
N1
left right
N2
root
N0
null
left right
T0
this.repOk()==false (increase index)
this.root
N0.left
N0.right
N0.left
![Page 44: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/44.jpg)
N0 N1 null null null null null
this
root left right
N0
left right
N1
left right
N2
root
N0
null
left right
N1
T0
this.repOk()==true (move right)
this.root
N0.left
N0.right
N1.left
N1.right
![Page 45: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/45.jpg)
N0 N1 N0 null null null null
this
root left right
N0
left right
N1
left right
N2
root
N0 left right
N1
T0
this.repOk()==false (increase index)
this.root
N0.left
N0.right
![Page 46: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/46.jpg)
N0 N1 N1 null null null null
this
root left right
N0
left right
N1
left right
N2
root
N0 left right
N1
T0
this.repOk()==false (increase index)
this.root
N0.left
N0.right
![Page 47: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/47.jpg)
N0 N1 N2 null null null null
this
root left right
N0
left right
N1
left right
N2
root
N0 left right
N1 N2
T0
this.repOk()==true (backtrack)
this.root
N0.left
N0.right
N1.left
N1.right
N2.left
N2.right
![Page 48: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/48.jpg)
N0 null N0 null null null null
this
root left right
N0
left right
N1
left right
N2
root
N0 left right
null
T0
this.repOk()==true (move right)
this.root
N0.left
N0.right
![Page 49: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/49.jpg)
N0 null N1 null null null null
this
root left right
N0
left right
N1
left right
N2
root
N0 left right
N1 null
T0
null null
left right
this.repOk()==true (move right)
this.root
N0.left
N0.right
N1.left
N1.right
![Page 50: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/50.jpg)
N0 null N1 N0 null null null
this
root left right
T0
N0
left right
N1
left right
N2
root
N0 left right
N1 null
left null
right
this.repOk()==false (increase index)
this.root
N0.left
N0.right
N1.left
![Page 51: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/51.jpg)
N0 null N1 N1 null null null
this
root left right
T0
N0
left right
N1
left right
N2
root
N0 left right
N1 null
left null
right
this.repOk()==false (increase index)
this.root
N0.left
N0.right
N1.left
![Page 52: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/52.jpg)
N0 null N1 N2 null null null
this
root left right
T0
N0
left right
N1
left right
N2
root
N0 left right
N1 null
left
null
right
N2
this.repOk()==true (backtrack)
this.root
N0.left
N0.right
N1.left
N1.right
N2.left
N2.right
![Page 53: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/53.jpg)
![Page 54: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/54.jpg)
Isomorphic instances enumeration: Given all the already assigned values, Korat can only use the same Restriction on values (only previous values +1)
Prune un-‐interesting search space: Korat monitors the field accesses to judge which fields must be modified during backtracking.
![Page 55: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/55.jpg)
For each non-‐isomorphic instance up to the chosen finitization satisfying the Java predicate
Creates the instance using Java reflection
Execute the selected method on the instance
Check the Java predicates on exit
![Page 56: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/56.jpg)
How do we deal with Strings, floats, etc.?
Korat does not build the instance using a method sequence
The performance of the Java predicate directly affects the Test Case Generation process.
Exhaustive generation can be infeasible for bigger scopes (100? 1000? 100000?)
![Page 57: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/57.jpg)
Randoop Random Testing with Feedback loop
Korat Efficient non-‐isomorphic test input generation
How can we use the source code to generate inputs? Pex: white-‐box test case generation using concolic execution.
![Page 58: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/58.jpg)
Program +
Specification Translator
Logical Formula
Constraint Solver Translator
Examples: ESC/Java2, JMLForge
![Page 59: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/59.jpg)
//@ requires …!//@ ensures …!procedure m(x,y,z) {! if (x>y) {!
!…! } else {!
!…! }! if (z==y) {!
!…! }!}!!!
Constraint Solver Translator
![Page 60: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/60.jpg)
Incremental solvers (SMT/SAT) allow the user to add new clauses/axioms when a solution is produced This could be use to enumerate more solutions Or it could be use to enumerate “interesting” executions of the program
![Page 61: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/61.jpg)
//@ requires …!//@ ensures false;!procedure m(x,y,z) {! if (x>y) {!
!GOAL_0!!…!
} else {!!GOAL_1!!…!
}! if (z==y) {!
!GOAL_2!!…!
} else {! GOAL_3! }!}!!!
Incremental Constraint Solver
Translator
![Page 62: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/62.jpg)
Incremental Constraint Solver
Translator
Program +
Goals SAT x=1 y=0 z=0
Counterexample: an execution trace (inputs) s.t. the postcondition fails (ensures “false”)
some goals were covered
![Page 63: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/63.jpg)
Incremental Constraint Solver
Translator
Program +
Goals SAT x=1 y=0 z=0
(Goals 0 && 2 covered)
we push a new axiom to the Constraint solver Goal 1 OR Goal 2 should be covered now
Test Inputs 1) x=1 y=0 z=0
![Page 64: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/64.jpg)
Incremental Constraint Solver
____________ goal 1 or goal3 ____________
Translator
Program +
Goals SAT x=9 y=9 z=9
(Goals 1 && 2 covered)
we push a new axiom to the Constraint solver Goal 1 OR Goal 3 should be covered now
Test Inputs 1) x=1 y=0 z=0
2) x=9 y=9 z=9
![Page 65: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/65.jpg)
Incremental Constraint Solver
____________ goal 0 or goal2 ____________
goal3
Translator
Program +
Goals SAT x=15 y=-‐2 z=-‐4
(Goals 0 && 3 covered)
we push a new axiom to the Constraint solver Goal 3 should be covered now
Test Inputs 1) x=1 y=0 z=0
2) x=9 y=9 z=9
3) x=15 y=-‐2 z=-‐4
![Page 66: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/66.jpg)
Incremental Constraint Solver
____________ goal 0 or goal2 ____________
goal3
Translator
Program +
Goals SAT x=15 y=-‐2 z=-‐4
(Goals 0 && 3 covered)
No more goals to cover!
Test Inputs 1) x=1 y=0 z=0
2) x=9 y=9 z=9
3) x=15 y=-‐2 z=-‐4
![Page 67: Automatic!Test!Generation! · 2018-04-05 · Goals!of!a!Test!Suite! All!instructions! Allconditions(decisions)! Allpaths Allinputs Goals!might!not!be!feasible! Example:deadcode! Detection!of!unfeasible!goals!is](https://reader034.vdocument.in/reader034/viewer/2022050512/5f9cec450781743cd0575831/html5/thumbnails/67.jpg)
JML input programs Branch/Goal/Path coverage Incremental SAT-‐Solving http://www.dc.uba.ar/fajita
Advantages Fewer calls to the constraint solver in worst case
Disadvantages Complete CFG has to be encoded as a formula (more complex that only a path)