test input generation for java containers using state matching willem visser corina pasareanu and...

Test Input Generation for Java Containers using

State Matching

Willem Visser Corina Pasareanu and Radek Pelanek

Automated Software Engineering GroupNASA Ames Research Center

Background• ISSTA 2004

– White-box Test Input Generation for Red-Black Trees• Symbolic Execution of structural invariants

– Korat-style

• Symbolic Execution of actual code

– Verdict• Got branch coverage for structures of < 7 nodes

– Could not go much higher either

• Satisfactory results, but is this how one would really test containers?

Background (2)• More natural to test using Black-box techniques

– Test cases based on API calls and parameters

• Additional inspiration from Paolo Tonella– Email exchange on comparing with his evolutionary

testing framework that works at the API level

• But there are many ways to do test generation at the API level…which way should we go– As many ways as we can!

• How to compare these?– Measure the coverage we get

General Idea

SUT

ENV (m,n)

m is the seq. length of API calls &

n is the number of values used in the parameters of the calls

API…

put(v)del(v)

Evaluate different techniques for selecting

test-cases from ENV(m,n)to obtain maximum coverage

SUT – Container Classes

• Binary Tree– 154 LOC– add(x), remove(x)

• Red-black Trees (java.util.TreeMap)– 580 LOC– put(x), remove(x)

• Fibonacci Heap– 286 LOC– Insert(x), delete(x), removeMin()

• Binomial Heap– 355 LOC– insert(x), delete(x), extractMin(), decreaseKey(x,y)

Coverage Measures

• Basic Block, i.e. Statement, Coverage– Simplest form of coverage– Aligned with “simple” bugs

• Predicate Coverage– Cover all combinations of a given set of predicates at

each branch in the code– Much harder coverage to obtain– Aligned with “deeper” behavioral bugs

• Question: why not seed bugs?– Only automatic way of doing this is through mutation

which is more likely to give easy bugs to find

Predicate Coverage

Cover all combinations of a given set of predicates at each branch in the code

Red-Black Tree Predicatesroot = null, e.left = null,

e.right = null, e.parent = null, e.color = BLACK

public void add(int x) {Node current = root;

if (root == null) {gen(0, current, x, null, null);root = new Node(x);return;

}

while (current.value != x) {if (x < current.value) {

if (current.left == null) {gen(1, current, x, null, null);current.left = new Node(x);

} else {gen(2, current, x, null, null);current = current.left;

}} else {

if (current.right == null) {gen(3, current, x, null, null);current.right = new Node(x);

} else {gen(4, current, x, null, null);current = current.right;

}}

}}

public static int gen_native(MJIEnv env, int objRef, int br, int n0, int x,int n1, int n2) {

String res = br + ",";

int temp;if (n0 == -1) {

res += "-";} else {

temp = env.getIntField(n0, null, "left");res += (temp == -1) ? "L-" : "L+";temp = env.getIntField(n0, null, "right");res += (temp == -1) ? "R-" : "R+";

}res += (n1 == -1) ? "P-" : "P+";if (n2 == -1) {

res += "B-";} else {

temp = env.getIntField(n2, null, "left");res += (temp == -1) ? "BL-" : "BL+";temp = env.getIntField(n2, null, "right");res += (temp == -1) ? "BR-" : "BR+";

}

if (!tests.contains(res)) {tests.add(res);System.out.println("Test case number " + tests.size() + " for '"

+ res + "': ");return tests.size();

}return 0;

}

Let’s Go Outside the Box

• Rather than over-approximate and refine, we under-approximate and refine– Clearly complements existing techniques

• If we restrict ourselves only to feasible behaviors when under-approximating then all safety property violations will be preserved

• Build on top of classic explicit-state model checking infrastructure

Classic Explicit-State SearchPROCEDURE dfs() { s = top(Stack); FOR all transitions t enabled in s DO s' = successor(s) after executing t; IF s' NOT IN VisitedStates THEN

Enter s' into VisitedStates; Push s' onto Stack; dfs(); END END; Pop s from Stack; }

INIT { Enter s0 into VisitedStates; Push s0 onto Stack; dfs(); }

Explicit-State (1-step) αSearchPROCEDURE dfs() { s = top(Stack); FOR all transitions t enabled in s DO s' = successor(s) after executing t; IF α(s‘) NOT IN VisitedStates THEN

Enter α(s‘) into VisitedStates; Push s' onto Stack; dfs(); END END; Pop s from Stack; }

INIT { Enter α(s0) into VisitedStates; Push s0 onto Stack; dfs(); }

αSearch

1: x = 2;2: while (x>0) 3: x = x - 1;4: assert false;

Abstraction Mapping

p = (x>0)

Map concrete states to abstract states for state storing

1,p

2,p

3,p

Always traverse only feasible paths

Under-approximation of the behaviors

Symbolic Execution and αSearch

• Current implementation is for a simple input language – oCaml using Simplify as a decision procedure

• We would like to integrate the technique in Java Pathfinder (JPF) that supports symbolic execution (using the Omega Library)– To allow application to programs with complex data

structures (objects)• Idea

– Execute symbolically but along concrete path– Whenever the symbolic analysis can follow

alternatives, add predicates for the alternative path

End of Part One

• Showed under-approximation based search with refinement– Backward weakest precondition based– Suggested forward symbolic execution based

• Part Two– Rather than automated refinement we use

user-provided abstractions– Motivation is to generate test-cases to

achieve high behavioral coverage for Java container classes

Explicit-State (1-step) αSearchPROCEDURE dfs() { s = top(Stack); FOR all transitions t enabled in s DO s' = successor(s) after executing t; IF α(s‘) NOT IN VisitedStates THEN

Enter α(s‘) into VisitedStates; Push s' onto Stack; dfs(); END END; Pop s from Stack; }

INIT { Enter α(s0) into VisitedStates; Push s0 onto Stack; dfs(); }

Techniques Considered• Random selection• Classic model checking

– State matching on complete state

• Abstraction search– State matching on abstract (partial) state

• Symbolic Execution– Complete matching using subsumption checks– Abstract matching

Framework

SUTwith

minor instrumentation

ENV

TestListener

Abstraction Mapping+

State StorageCoverage Manager

JPF

Sample Output

Test case number 77 for '15,L+R+P-REDroot': put(0);put(4);put(5);put(1);put(2);put(3);remove(4);

Unique ID for the testBranch Number Predicate Values

Test-case to achieve above coverage

Test case number 7 for '32,L-R-P+RED':X2(0) == X1(0) && X2(0) < X0(1) && X1(0) < X0(1)put(X0);put(X1);remove(X2);

Test case number 7 for '32,L-R-P+RED': put(1);put(0);remove(0);

Concrete

Symbolic

Path Conditionwith solutions

Symbolic TC

Environment SkeletonM : sequence lengthN : parameter valuesA : abstraction used

for (int i = 0; i < M; i++) { int x = Verify.random(N - 1); switch (Verify.random(1)) { case 0: put(x); break; case 1: remove(x); break;} }Verify.ignoreIf(checkAbstractState(A));

Symbolic Environment Skeleton

M : sequence lengthA : abstraction used

for (int i = 0; i < M; i++) {

SymbolicInteger x = new SymbolicInteger(“X“+i); switch (Verify.random(1)) { case 0: put(x); break; case 1: remove(x); break;} }Verify.ignoreIf(checkAbstractState(A));

Abstraction Search

• Map state to an abstract version and backtrack if the abstract state was seen before, i.e. discard test-case

• Mapping can be lossy or not• Abstraction mappings can be created by

the user/tester• Default abstraction mappings are provided

Default Mappings

• Structure of the heap of the program– e.g. structure of the containers

• Structure augmented with non-data fields• Structure augmented with symbolic

constraints on the data in the structure– This requires checking constraint

subsumption

Linearization Comparing Structures

1

2

3 4

5

1

2

3 4

5

1

2

3 4

5

1

2

3 4

5

1 2 3 -1 -1 4 -1 -1 5 -1 -1 1 2 3 -1 -1 4 -1 -1 5 -1 -1

1 2 3 -1 -1 4 -1 -1 5 -1 -1 1 2 3 -1 -1 4 -1 5 -1 -1 -1

4 3

2

1

Linearization + Mapping

1

2

3 4

5

1b 2b 3r -1 -1 4r -1 -1 5b -1 -1

5

1b 2r 3r -1 -1 4r -1 -1 5r -1 -1

Linearization takes a mapping object as parameter to indicate how each node in the heap should be linearized.

In the example above each node gets, besides the unique identifier, a mapping of “r” if the original structure had a red node and “b” if the original structure had a black node in that position. If we also added the key values for each node the linearization might have looked something like: 1b6 2b4 3r3 -1 -1 4r5 -1 -1 5b7 -1 -1

Symbolic Execution

Symbolic Statex1

x2

x3 x4

x5 +x1 > x2 & x2 > x3 & x2 < x4 & x5 > x1

Shape Symbolic Constraints

Subsumption Checking

x1

x2

x3 x4

x5 +x1 > x2 & x2 > x3 & x2 < x4 & x5 > x1

x1

x2

x3 x4

x5 +x1 > x2 & x2 > x3 & x2 < x4 & x5 > x1

If only it was this simple!

Getting Ready for CheckingExistential Elimination

x1

x2

x3 x4

x5

PCs1 < s2 & s4 > s3 & s4 < s1 & s4 < s5 &s7 < s2 & s7 > s1

s1

s4 s2

s3 s5

+

s1,s2,s3,s4,s5 such that x1 = s1 & x2 = s4 & x3 = s3 & x4 = s5 & x5 = s2 & PC

x1 > x2 & x2 > x3 & x2 < x4 & x5 > x1

Bidirectional Subsumption Checking

• If new => old – backtrack

• If old => new – new is more general than old – replace old with new

• to increase chances of getting a match in the future – Continue on path from new, i.e. don’t backtrack

• Ultimately for each shape we want to use disjunction of constraints– Small technicality prevents us – bug in omega lib

Evaluation

• Red-Black Trees• Out of Memory runs are not reported• Breadth-first Search unless stated• Sequence Length = Values

for the non-symbolic searches• First compare under Branch Coverage

Exhaustive TechniquesBranch Coverage

Seq Cov Len Time MemFull MC 7 39 4.3 536 584

S+C+V 7 39 4.3 10.635 17.47

Sym – S+Sub 7 39 4.3 14.201 16.95

Optimal Branch Coverage is 39

Under-Approximation TechniquesBranch Coverage

Seq Cov Len Time MemS 21 39 6.1 57.353 72.07

S+C 18 39 5.8 32.577 21.16

Sym - S 7 39 4.3 10.054 15.43

Sym – S+C 7 39 4.3 11.998 10.76

Random 9 39 7 40.429 3.06

Optimal Branch Coverage is 39

Exhaustive TechniquesPredicate Coverage

Seq Cov Len Time MemFull MC 7 79 5.2 543 309

S+C+V 10 95 5.7 350 228

Sym – S+Sub 11 102 6.1 222 117

Optimal Predicate Coverage is 106

Under-Approximation TechniquesPredicate Coverage

Seq Cov Len Time MemS 25d 106 21.7 90 13.31

S+C 30 106 8.3 354 100

Sym - S 12 100 6.1 230 123.27

Sym – S+C 12 104 6.2 356 138

Random 60 106 30.1 61.459 7.74

Optimal Predicate Coverage is 106

Observations

• For a simple coverage such as branch coverage, all the techniques work well, including the exhaustive ones

• But making the coverage more “behavioral”, even by a small increment, kills off the exhaustive techniques

Observations

• Full Blown Model Checking doesn’t work here• Its close cousin, that only looks at the relevant

state at the relevant time, scales much better• Branch - full coverage after:

– MC: 536s & 584Mb– Complete: 10s & 17Mb

• Predicate – best coverage after:– MC: 79 covered with 543s & 309Mb– Complete: 95 covered with 350s & 228Mb

Observations

• Symbolic techniques have a slight edge over concrete ones for exhaustive analysis– Comparing for Predicate Coverage (10)

• Full Concrete(95): 350s & 228Mb• Full Symbolic(95): 123s & 62Mb

• Current results indicate symbolic under-approximation based search is less efficient than concrete

• Further experimentation required

Observations

• Random Search?– Seems to work rather well here

• It will always have an edge on memory, since it uses almost none

• It will most likely have an edge on speed, since it needs to do little additional work – it will however redo work often

• It will in general do worse on test-case length, since it requires longer sequences to achieve more complex coverage

Observations

• Search Order Matters for the lossy techniques– BFS is inherently better than DFS– On occasion though it is the other way round

Conclusions & Future Work

• Showed how predicate abstraction can be used for an under-approximation based search with refinement

• Showed how a lightweight variant, where the abstraction mapping is given and no refinement is done, can be used for bug-finding and test-case generation

• Goal: Derive predicates for analyzing containers automatically through the use of symbolic execution during refinement– Can we derive shape predicates automatically?