Test Input Generation for Java Containers using

State Matching

Willem Visser Corina Pasareanu and Radek Pelanek

Automated Software Engineering GroupNASA Ames Research Center

Background• ISSTA 2004

– White-box Test Input Generation for Red-Black Trees• Symbolic Execution of structural invariants

– Korat-style

• Symbolic Execution of actual code

– Verdict• Got branch coverage for structures of < 7 nodes

– Could not go much higher either

• Satisfactory results, but is this how one would really test containers?

Background (2)• More natural to test using Black-box techniques

– Test cases based on API calls and parameters

• Additional inspiration from Paolo Tonella– Email exchange on comparing with his evolutionary

testing framework that works at the API level

• But there are many ways to do test generation at the API level…which way should we go– As many ways as we can!

• How to compare these?– Measure the coverage we get

General Idea

SUT

ENV (m,n)

m is the seq. length of API calls &

n is the number of values used in the parameters of the calls

API…

put(v)del(v)

Evaluate different techniques for selecting

test-cases from ENV(m,n)to obtain maximum coverage

SUT – Container Classes

• Binary Tree– 154 LOC– add(x), remove(x)

• Red-black Trees (java.util.TreeMap)– 580 LOC– put(x), remove(x)

• Fibonacci Heap– 286 LOC– Insert(x), delete(x), removeMin()

• Binomial Heap– 355 LOC– insert(x), delete(x), extractMin(), decreaseKey(x,y)

Coverage Measures

• Basic Block, i.e. Statement, Coverage– Simplest form of coverage– Aligned with “simple” bugs

• Predicate Coverage– Cover all combinations of a given set of predicates at

each branch in the code– Much harder coverage to obtain– Aligned with “deeper” behavioral bugs

• Question: why not seed bugs?– Only automatic way of doing this is through mutation

which is more likely to give easy bugs to find

Predicate Coverage

Cover all combinations of a given set of predicates at each branch in the code

Red-Black Tree Predicatesroot = null, e.left = null,

e.right = null, e.parent = null, e.color = BLACK

public void add(int x) {Node current = root;

if (root == null) {gen(0, current, x, null, null);root = new Node(x);return;

}

while (current.value != x) {if (x < current.value) {

if (current.left == null) {gen(1, current, x, null, null);current.left = new Node(x);

} else {gen(2, current, x, null, null);current = current.left;

}} else {

if (current.right == null) {gen(3, current, x, null, null);current.right = new Node(x);

} else {gen(4, current, x, null, null);current = current.right;

}}

}}

public static int gen_native(MJIEnv env, int objRef, int br, int n0, int x,int n1, int n2) {

String res = br + ",";

int temp;if (n0 == -1) {

res += "-";} else {

temp = env.getIntField(n0, null, "left");res += (temp == -1) ? "L-" : "L+";temp = env.getIntField(n0, null, "right");res += (temp == -1) ? "R-" : "R+";

}res += (n1 == -1) ? "P-" : "P+";if (n2 == -1) {

res += "B-";} else {

temp = env.getIntField(n2, null, "left");res += (temp == -1) ? "BL-" : "BL+";temp = env.getIntField(n2, null, "right");res += (temp == -1) ? "BR-" : "BR+";

}

if (!tests.contains(res)) {tests.add(res);System.out.println("Test case number " + tests.size() + " for '"

+ res + "': ");return tests.size();

}return 0;

}

Let’s Go Outside the Box

• Rather than over-approximate and refine, we under-approximate and refine– Clearly complements existing techniques

• If we restrict ourselves only to feasible behaviors when under-approximating then all safety property violations will be preserved

• Build on top of classic explicit-state model checking infrastructure

Classic Explicit-State SearchPROCEDURE dfs() { s = top(Stack); FOR all transitions t enabled in s DO s' = successor(s) after executing t; IF s' NOT IN VisitedStates THEN

Enter s' into VisitedStates; Push s' onto Stack; dfs(); END END; Pop s from Stack; }

INIT { Enter s0 into VisitedStates; Push s0 onto Stack; dfs(); }

Explicit-State (1-step) αSearchPROCEDURE dfs() { s = top(Stack); FOR all transitions t enabled in s DO s' = successor(s) after executing t; IF α(s‘) NOT IN VisitedStates THEN

Enter α(s‘) into VisitedStates; Push s' onto Stack; dfs(); END END; Pop s from Stack; }

INIT { Enter α(s0) into VisitedStates; Push s0 onto Stack; dfs(); }

αSearch

1: x = 2;2: while (x>0) 3: x = x - 1;4: assert false;

Abstraction Mapping

p = (x>0)

Map concrete states to abstract states for state storing

1,p

2,p

3,p

Always traverse only feasible paths

Under-approximation of the behaviors

Symbolic Execution and αSearch

• Current implementation is for a simple input language – oCaml using Simplify as a decision procedure

• We would like to integrate the technique in Java Pathfinder (JPF) that supports symbolic execution (using the Omega Library)– To allow application to programs with complex data

structures (objects)• Idea

– Execute symbolically but along concrete path– Whenever the symbolic analysis can follow

alternatives, add predicates for the alternative path

End of Part One

• Showed under-approximation based search with refinement– Backward weakest precondition based– Suggested forward symbolic execution based

• Part Two– Rather than automated refinement we use

user-provided abstractions– Motivation is to generate test-cases to

achieve high behavioral coverage for Java container classes

Explicit-State (1-step) αSearchPROCEDURE dfs() { s = top(Stack); FOR all transitions t enabled in s DO s' = successor(s) after executing t; IF α(s‘) NOT IN VisitedStates THEN

Enter α(s‘) into VisitedStates; Push s' onto Stack; dfs(); END END; Pop s from Stack; }

INIT { Enter α(s0) into VisitedStates; Push s0 onto Stack; dfs(); }

Techniques Considered• Random selection• Classic model checking

– State matching on complete state

• Abstraction search– State matching on abstract (partial) state

• Symbolic Execution– Complete matching using subsumption checks– Abstract matching

Framework

SUTwith

minor instrumentation

ENV

TestListener

Abstraction Mapping+

State StorageCoverage Manager

JPF

Sample Output

Test case number 77 for '15,L+R+P-REDroot': put(0);put(4);put(5);put(1);put(2);put(3);remove(4);

Unique ID for the testBranch Number Predicate Values

Test-case to achieve above coverage

Test case number 7 for '32,L-R-P+RED':X2(0) == X1(0) && X2(0) < X0(1) && X1(0) < X0(1)put(X0);put(X1);remove(X2);

Test case number 7 for '32,L-R-P+RED': put(1);put(0);remove(0);

Concrete

Symbolic

Path Conditionwith solutions

Symbolic TC

Environment SkeletonM : sequence lengthN : parameter valuesA : abstraction used

for (int i = 0; i < M; i++) { int x = Verify.random(N - 1); switch (Verify.random(1)) { case 0: put(x); break; case 1: remove(x); break;} }Verify.ignoreIf(checkAbstractState(A));

Symbolic Environment Skeleton

M : sequence lengthA : abstraction used

for (int i = 0; i < M; i++) {

SymbolicInteger x = new SymbolicInteger(“X“+i); switch (Verify.random(1)) { case 0: put(x); break; case 1: remove(x); break;} }Verify.ignoreIf(checkAbstractState(A));

Abstraction Search

• Map state to an abstract version and backtrack if the abstract state was seen before, i.e. discard test-case

• Mapping can be lossy or not• Abstraction mappings can be created by

the user/tester• Default abstraction mappings are provided

Default Mappings

• Structure of the heap of the program– e.g. structure of the containers

• Structure augmented with non-data fields• Structure augmented with symbolic

constraints on the data in the structure– This requires checking constraint

subsumption

Linearization Comparing Structures

1

2

3 4

5

1

2

3 4

5

1

2

3 4

5

1

2

3 4

5

1 2 3 -1 -1 4 -1 -1 5 -1 -1 1 2 3 -1 -1 4 -1 -1 5 -1 -1

1 2 3 -1 -1 4 -1 -1 5 -1 -1 1 2 3 -1 -1 4 -1 5 -1 -1 -1

4 3

2

1

Linearization + Mapping

1

2

3 4

5

1b 2b 3r -1 -1 4r -1 -1 5b -1 -1

5

1b 2r 3r -1 -1 4r -1 -1 5r -1 -1

Linearization takes a mapping object as parameter to indicate how each node in the heap should be linearized.

In the example above each node gets, besides the unique identifier, a mapping of “r” if the original structure had a red node and “b” if the original structure had a black node in that position. If we also added the key values for each node the linearization might have looked something like: 1b6 2b4 3r3 -1 -1 4r5 -1 -1 5b7 -1 -1

Symbolic Execution

Symbolic Statex1

x2

x3 x4

x5 +x1 > x2 & x2 > x3 & x2 < x4 & x5 > x1

Shape Symbolic Constraints

Subsumption Checking

x1

x2

x3 x4

x5 +x1 > x2 & x2 > x3 & x2 < x4 & x5 > x1

x1

x2

x3 x4

x5 +x1 > x2 & x2 > x3 & x2 < x4 & x5 > x1

If only it was this simple!

Getting Ready for CheckingExistential Elimination

x1

x2

x3 x4

x5

PCs1 < s2 & s4 > s3 & s4 < s1 & s4 < s5 &s7 < s2 & s7 > s1

s1

s4 s2

s3 s5

+

s1,s2,s3,s4,s5 such that x1 = s1 & x2 = s4 & x3 = s3 & x4 = s5 & x5 = s2 & PC

x1 > x2 & x2 > x3 & x2 < x4 & x5 > x1

Bidirectional Subsumption Checking

• If new => old – backtrack

• If old => new – new is more general than old – replace old with new

• to increase chances of getting a match in the future – Continue on path from new, i.e. don’t backtrack

• Ultimately for each shape we want to use disjunction of constraints– Small technicality prevents us – bug in omega lib

Evaluation

• Red-Black Trees• Out of Memory runs are not reported• Breadth-first Search unless stated• Sequence Length = Values

for the non-symbolic searches• First compare under Branch Coverage

Exhaustive TechniquesBranch Coverage

Seq Cov Len Time MemFull MC 7 39 4.3 536 584

S+C+V 7 39 4.3 10.635 17.47

Sym – S+Sub 7 39 4.3 14.201 16.95

Optimal Branch Coverage is 39

Under-Approximation TechniquesBranch Coverage

Seq Cov Len Time MemS 21 39 6.1 57.353 72.07

S+C 18 39 5.8 32.577 21.16

Sym - S 7 39 4.3 10.054 15.43

Sym – S+C 7 39 4.3 11.998 10.76

Random 9 39 7 40.429 3.06

Optimal Branch Coverage is 39

Exhaustive TechniquesPredicate Coverage

Seq Cov Len Time MemFull MC 7 79 5.2 543 309

S+C+V 10 95 5.7 350 228

Sym – S+Sub 11 102 6.1 222 117

Optimal Predicate Coverage is 106

Under-Approximation TechniquesPredicate Coverage

Seq Cov Len Time MemS 25d 106 21.7 90 13.31

S+C 30 106 8.3 354 100

Sym - S 12 100 6.1 230 123.27

Sym – S+C 12 104 6.2 356 138

Random 60 106 30.1 61.459 7.74

Optimal Predicate Coverage is 106

Observations

• For a simple coverage such as branch coverage, all the techniques work well, including the exhaustive ones

• But making the coverage more “behavioral”, even by a small increment, kills off the exhaustive techniques

Observations

• Full Blown Model Checking doesn’t work here• Its close cousin, that only looks at the relevant

state at the relevant time, scales much better• Branch - full coverage after:

– MC: 536s & 584Mb– Complete: 10s & 17Mb

• Predicate – best coverage after:– MC: 79 covered with 543s & 309Mb– Complete: 95 covered with 350s & 228Mb

Observations

• Symbolic techniques have a slight edge over concrete ones for exhaustive analysis– Comparing for Predicate Coverage (10)

• Full Concrete(95): 350s & 228Mb• Full Symbolic(95): 123s & 62Mb

• Current results indicate symbolic under-approximation based search is less efficient than concrete

• Further experimentation required

Observations

• Random Search?– Seems to work rather well here

• It will always have an edge on memory, since it uses almost none

• It will most likely have an edge on speed, since it needs to do little additional work – it will however redo work often

• It will in general do worse on test-case length, since it requires longer sequences to achieve more complex coverage

Observations

• Search Order Matters for the lossy techniques– BFS is inherently better than DFS– On occasion though it is the other way round

Conclusions & Future Work

• Showed how predicate abstraction can be used for an under-approximation based search with refinement

• Showed how a lightweight variant, where the abstraction mapping is given and no refinement is done, can be used for bug-finding and test-case generation

• Goal: Derive predicates for analyzing containers automatically through the use of symbolic execution during refinement– Can we derive shape predicates automatically?