austin: a tool for search based software testing and its evaluation on deployed automotive systems...
TRANSCRIPT
AUSTIN: A TOOL FOR SEARCH BASED SOFTWARE TESTINGAnd its Evaluation on Deployed Automotive Systems
Kiran Lakhotia, Mark Harman and Hamilton Gross in
Publication Trend in SBSE
Spread of Activities in SBSE
Yet Another Search Based Testing Tool?
IGUANA - for C (Phil
McMinn)
eTOC – for JAVA (Paolo
Tonella)
Others?
Yet Another Search Based Testing Tool?
IGUANA - for C (Phil McMinn)
eTOC – for JAVA
(Paolo Tonella)
Others?
AUSTIN
Outline
AUSTIN •Introduction to Austin
ETF •Overview of ETF
EmpiricalStudy
•Hypotheses, Test Subjects•Results and Analysis
Conclusion •ETF or AUSTIN?
AUgmented Search based TestINg
CIL
AST, CFG
instrument
gcc -o sut.exe
runAUSTIN
gcc -P -Ec sourcefiles
preprocessed
c sourcefiles
instrumented
c source
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 87, key2 = 33, p = 0, q = 0
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 88, key2 = 33, p = 0, q = 0
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 86, key2 = 33, p = 0, q = 0
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 84, key2 = 33, p = 0, q = 0
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 80, key2 = 33, p = 0, q = 0
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 80, key2 = 33, p = 0, q = 0
Fitness Function:approach_level + norm(branch_distance)
Search Method:Alternating Variable Method
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0, q = 0
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0, q = 0
Symbolic Path Condition:key1 == key2 && p == 0
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0, q = 0
Symbolic Path Condition:key1 == key2 && p == 0
Simplified Path Condition:p == 0
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0, q = 0
Symbolic Path Condition:key1 == key2 && p == 0
Simplified Path Condition:p == 0
Solve: p ≠ 0
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0, q = 0
Symbolic Path Condition:key1 == key2 && p == 0
Simplified Path Condition:p == 0
Solve: p ≠ 0
p 0
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0x..., q = 0
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0x..., q = 0
Symbolic Path Condition:key1 == key2 && p ≠ 0 && p ≠ q
Simplified Path Condition:p ≠ 0 && p ≠ q
Solve: p ≠ 0 && p == qp 0
q
p,q
0
AUgmented Search based TestINg
void testme(key1, key2, p, q)int key1;int key2;int* p;int* q; { if(key1 == key2) if(p != 0) if(p == q) //target}
Inputs:key1 = 33, key2 = 33, p = 0x..., q = 0x...
Evolutionary Testing Framework (ETF)
rZSlJ3hb
ETF was developed as part of the EvoTest project Integrated into the Eclipse IDE Supports white & black box testing Uses Evolutionary Algorithms (through the
GUIDE, EO evolutionary library interface) to generate test data
ETF
Selection
Recombinatio
n
Mutation
ReinsertionGUIDE + EO
Send Fitness Values
Send Individuals
ETF cont.rZSlJ3hb
Screenshots taken from a talk by Peter M. Kruse from Berner & Mattner, April 2009
ETF cont.rZSlJ3hb
Screenshots taken from a talk by Peter M. Kruse from Berner & Mattner, April 2009
Fitness Function and Algorithm Both ETF and AUSTIN use the standard
fitness function for branch coverage in Search Based Testing apporach_level + norm(branch_distance)
normalization function: 1 - 1.001-branch_distance
ETF uses a Genetic Algorithm AUSTIN uses the Alternating Variable
Method
Pointers in ETF and AUSTIN
AUSTIN Construct an equivalence of memory locations based on =
and ≠ operators in symbolic path condition Initialize (pointer) input variables based on the graph
Assign new memory location via malloc or use existing memory location
On-demand pointer handling ETF
Generate an index and auxiliary variable for each pointer input
Assign all input variables of the same type (including globals) to a pool Optimize indices and values of auxiliary variables along with the
inputs. Indices are used to select variables from pools and assign their address to the pointer input
Pointers in ETF
pool of input parameters and auxiliary variables
pCell
globDbl
val_p, val_q,globX
struct cell{ int value; struct cell* next;};int globX;double globDbl;struct cell* pCell;
void testme(int* p, int* q){ ...}
p = &globX;q = &val_p;
Hypothesis
Null Hypotheses Alternate Hypotheses
AUSTIN is as effective as the ETF in achieving branch coverage.
AUSTIN is more effective than the ETF in achieving branch coverage.
AUSTIN is equally as efficient as the ETF in achieving branch coverage of a function.
AUSTIN is more efficient than the ETF in achieving branch coverage of a function.
Empirical Study
3 case studies (comprising 8 functions) Embedded Software Modules from
Automotive IndustryCase Study
Description LOC - Tested
Total Branches
Nr. Inputs
B Adaptive Headlight Control Software
1,236 598 132
C Door-Lock Control Software
383 315 71
D Electric Windows Control Software
339 248 31
Coverage ETF vs AUSTIN
Efficiency ETF vs AUSTIN
Coverage Random vs AUSTIN
Efficiency Random vs AUSTIN
ETF or AUSTIN?
Coverage is about the same AUSTIN is overall a little bit more
efficient than the ETF in terms of fitness evaluations. In terms of wall clock time the difference is greater (due to the overhead of the ETF-GUIDE setup)
AUSTIN does not construct random memory graphs
ETF is not able generate inputs such as
struct cell{ int mem;};void testme(int* p, struct cell* c) { if( p == &c->mem ) //target}
Summary
Summary
http://www.cs.ucl.ac.uk/staff/K.Lakhotia/software/austin.htm