Download - Hybrid Concolic Testing
![Page 1: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/1.jpg)
Hybrid Concolic Testing
Rupak Majumdar Koushik SenUC Los Angeles UC Berkeley
![Page 2: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/2.jpg)
Automated Test Generation Studied since 70’s
King 76, Myers 79 30 years have passed, and yet no
effective solution What Happened???
![Page 3: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/3.jpg)
Automated Test Generation Studied since 70’s
King 76, Myers 79 30 years have passed, and yet no
effective solution What Happened???
Program-analysis techniques were expensive
Automated theorem proving and constraint solving techniques were not efficient
![Page 4: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/4.jpg)
Automated Test Generation Studied since 70’s
King 76, Myers 79 30 years have passed, and yet no
effective solution What Happened???
Program-analysis techniques were expensive
Automated theorem proving and constraint solving techniques were not efficient
In the recent years we have seen remarkable progress in static program-analysis and constraint solving SLAM, BLAST, ESP, Bandera, Saturn, MAGIC
![Page 5: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/5.jpg)
Automated Test Generation Studied since 70’s
King 76, Myers 79 30 years have passed, and yet no
effective solution What Happened???
Program-analysis techniques were expensive
Automated theorem proving and constraint solving techniques were not efficient
In the recent years we have seen remarkable progress in static program-analysis and constraint solving SLAM, BLAST, ESP, Bandera, Saturn, MAGIC
Question: Can we combine program analysis with
classical testing techniques to Scale Automated Test
Generation?
![Page 6: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/6.jpg)
Our Approach
Concolic Testing:
1.Combines Dynamic and Static Program Analysis
2.Exhaustive
3.Fails to scale
Random Testing:
1.Fast
2.Non-exhaustive
3.Redundant Executions and poor coverage
+
=Hybrid Concolic Testing
![Page 7: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/7.jpg)
Goals of Test Generation (Simplified) Generate test inputs Execute program on generated test
inputs Catch assertion violations Problem: how to ensure that all reachable
statements are executed Solution:
Explore all feasible execution paths
![Page 8: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/8.jpg)
Execution of Programs All Possible
Execution Paths Binary tree
Computation tree Internal node !
conditional statement execution
Edge ! execution of a sequence of non-conditional statements
Each path in the tree represents an equivalence class of inputs
F T
F F
F
F
T
T
T
T
T
T
Conditional Statements
Non-Conditional Statements
![Page 9: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/9.jpg)
Fuzz (Random) Testing Random testing
Random Testing [Bird and Munoz 83]
Fuzz testing Windows NT [Forrester and Miller 00]
QuickCheck [Claessen & Hughes 01]
JCrasher [Csallner and Smaragdakis 04]
RUTE-J [Andrews et al. 06] Randoop [Pacheco et al.
07]
Very low probability of reaching an error
Problematic for complex data structures
![Page 10: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/10.jpg)
Fuzz (Random) Testing Random testing
Random Testing [Bird and Munoz 83]
Fuzz testing Windows NT [Forrester and Miller 00]
QuickCheck [Claessen & Hughes 01]
JCrasher [Csallner and Smaragdakis 04]
RUTE-J [Andrews et al. 06] Randoop [Pacheco et al.
07]
Very low probability of reaching an error
Problematic for complex data structures
Example ( ) {
s = readString();
if (s[0]==‘I’ && s[1]==‘C’ &&
s[2]==‘S’ && s[3]==‘E’ &&
s[4]==‘2’ && s[5]==‘0’ &&
s[6]==‘0’ && s[7]==‘7’) {
printf(“Am I here?”);
}
}
Example ( ) {
s = readString();
if (s[0]==‘I’ && s[1]==‘C’ &&
s[2]==‘S’ && s[3]==‘E’ &&
s[4]==‘2’ && s[5]==‘0’ &&
s[6]==‘0’ && s[7]==‘7’) {
printf(“Am I here?”);
}
}
Input domain = {‘0’, ‘2’, ‘7’, ‘C’, ‘E’, ‘I’, ‘S’}
Probability of reaching printf = 7-8 » 10-7
![Page 11: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/11.jpg)
Fuzz (Random) Testing Random testing
Random Testing [Bird and Munoz 83]
Fuzz testing Windows NT [Forrester and Miller 00]
QuickCheck [Claessen & Hughes 01]
JCrasher [Csallner and Smaragdakis 04]
RUTE-J [Andrews et al. 06] Randoop [Pacheco et al.
07]
Very low probability of reaching an error
Problematic for complex data structures
Example ( ) {
s = readString();
if (s[0]==‘I’ && s[1]==‘C’ &&
s[2]==‘S’ && s[3]==‘E’ &&
s[4]==‘2’ && s[5]==‘0’ &&
s[6]==‘0’ && s[7]==‘7’) {
printf(“Am I here?”);
}
}
Example ( ) {
s = readString();
if (s[0]==‘I’ && s[1]==‘C’ &&
s[2]==‘S’ && s[3]==‘E’ &&
s[4]==‘2’ && s[5]==‘0’ &&
s[6]==‘0’ && s[7]==‘7’) {
printf(“Am I here?”);
}
}
Input domain = {‘0’, ‘2’, ‘7’, ‘C’, ‘E’, ‘I’, ‘S’}
Probability of reaching printf = 7-8 » 10-7 Fast and
Inexpensive
![Page 12: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/12.jpg)
Concolic Testing Combine concrete testing (concrete
execution) and symbolic testing (symbolic execution) [PLDI’05, FSE’05, FASE’06, CAV’06,
HVC’06]
Concrete + Symbolic = Concolic
![Page 13: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/13.jpg)
Example
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
![Page 14: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/14.jpg)
Example
ERROR
2*y == x
x > y+10
Y
Y
N
N
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
![Page 15: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/15.jpg)
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 22, y = 7 x = x0, y = y0
![Page 16: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/16.jpg)
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 22, y = 7, z = 14
x = x0, y = y0, z = 2*y0
![Page 17: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/17.jpg)
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 22, y = 7, z = 14
x = x0, y = y0, z = 2*y0
2*y0 != x0
![Page 18: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/18.jpg)
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
2*y0 != x0
Solve: 2*y0 == x0
Solution: x0 = 2, y0 = 1
x = 22, y = 7, z = 14
x = x0, y = y0, z = 2*y0
![Page 19: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/19.jpg)
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 2, y = 1 x = x0, y = y0
![Page 20: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/20.jpg)
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 2, y = 1, z = 2
x = x0, y = y0, z = 2*y0
![Page 21: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/21.jpg)
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 2, y = 1, z = 2
x = x0, y = y0, z = 2*y0
2*y0 == x0
![Page 22: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/22.jpg)
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 2, y = 1, z = 2
x = x0, y = y0, z = 2*y0
2*y0 == x0
x0 · y0+10
![Page 23: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/23.jpg)
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 2, y = 1, z = 2
x = x0, y = y0, z = 2*y0
Solve: (2*y0 == x0) Æ (x0 > y0 + 10)
Solution: x0 = 30, y0 = 15
2*y0 == x0
x0 · y0+10
![Page 24: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/24.jpg)
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 30, y = 15 x = x0, y = y0
![Page 25: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/25.jpg)
Concolic Testing Approach
int double (int v) {
return 2*v; }
void testme (int x, int y) {
z = double (y);
if (z == x) {
if (x > y+10) {
ERROR;}
}
}
Concrete Execution
Symbolic Execution
concrete state
symbolic state
path condition
x = 30, y = 15 x = x0, y = y0
2*y0 == x0
x0 > y0+10
Program Error
![Page 26: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/26.jpg)
Explicit Path (not State) Model Checking Traverse all
execution paths one by one to detect errors assertion violations program crash uncaught exceptions
combine with valgrind to discover memory errors
F T
F F
F
F
T
T
T
T
T
T
![Page 27: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/27.jpg)
Explicit Path (not State) Model Checking Traverse all
execution paths one by one to detect errors assertion violations program crash uncaught exceptions
combine with valgrind to discover memory errors
F T
F F
F
F
T
T
T
T
T
T
![Page 28: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/28.jpg)
Explicit Path (not State) Model Checking Traverse all
execution paths one by one to detect errors assertion violations program crash uncaught exceptions
combine with valgrind to discover memory errors
F T
F F
F
F
T
T
T
T
T
T
![Page 29: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/29.jpg)
Explicit Path (not State) Model Checking Traverse all
execution paths one by one to detect errors assertion violations program crash uncaught exceptions
combine with valgrind to discover memory errors
F T
F F
F
F
T
T
T
T
T
T
![Page 30: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/30.jpg)
Explicit Path (not State) Model Checking Traverse all
execution paths one by one to detect errors assertion violations program crash uncaught exceptions
combine with valgrind to discover memory errors
F T
F F
F
F
T
T
T
T
T
T
![Page 31: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/31.jpg)
Explicit Path (not State) Model Checking Traverse all
execution paths one by one to detect errors assertion violations program crash uncaught exceptions
combine with valgrind to discover memory errors
F T
F F
F
F
T
T
T
T
T
T
![Page 32: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/32.jpg)
Limitations Path Space of a Large Program is Huge
Path Explosion Problem
Entire Computation Tree
![Page 33: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/33.jpg)
Limitations Path Space of a Large Program is Huge
Path Explosion Problem
Explored by Concolic Testing
Entire Computation Tree
![Page 34: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/34.jpg)
Limitations: A Comparative View
Concolic: Broad, shallow
Random: Narrow, deep
![Page 35: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/35.jpg)
Limitations: ExampleExample ( ) {1: state = 0;2: while(1) {3: s = input();4: c = input();5: if(c==‘:’ && state==0) state=1;6: else if(c==‘\n’ && state==1) state=2;7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) {
COVER_ME:; } }}
Example ( ) {1: state = 0;2: while(1) {3: s = input();4: c = input();5: if(c==‘:’ && state==0) state=1;6: else if(c==‘\n’ && state==1) state=2;7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) {
COVER_ME:; } }}
Similar code in
•Text editors (vi)
•Parsers (lexer)
•Event-driven programs (GUI)
•Want to hit COVER_ME
•input() denotes external input
•Can be hit on an input sequence
s = “ICSE”
c : ‘:’ ‘\n’
![Page 36: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/36.jpg)
Limitations: ExampleExample ( ) {1: state = 0;2: while(1) {3: s = input();4: c = input();5: if(c==‘:’ && state==0) state=1;6: else if(c==‘\n’ && state==1) state=2;7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) {
COVER_ME:; } }}
Example ( ) {1: state = 0;2: while(1) {3: s = input();4: c = input();5: if(c==‘:’ && state==0) state=1;6: else if(c==‘\n’ && state==1) state=2;7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) {
COVER_ME:; } }}
•Pure random testing can get to
state = 2
But difficult to get ‘ICSE’ as a
Sequence
Probability 1/(28)6 » 3X10-15
•Conversely, concolic testing
can generate ‘ICSE’ but explores
many paths to get to state = 2
![Page 37: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/37.jpg)
Hybrid Concolic Testing Interleave Random Testing and Concolic Testing
to increase coverage
Motivated by similar idea used in VLSI design validation:
Ganai et al. 1999, Ho et al. 2000
![Page 38: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/38.jpg)
Hybrid Concolic Testing Interleave Random Testing and Concolic Testing
to increase coverage
while (not required coverage) {
while (not saturation)
perform random testing;
Checkpoint;
while (not increase in coverage)
perform concolic testing;
Restore;
}
![Page 39: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/39.jpg)
Hybrid Concolic Testing Interleave Random Testing and Concolic Testing
to increase coverage
while (not required coverage) {
while (not saturation)
perform random testing;
Checkpoint;
while (not increase in coverage)
perform concolic testing;
Restore;
}
Deep, broad search
Hybrid Search
![Page 40: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/40.jpg)
Hybrid Concolic Testing Random Phase
‘$’, ‘&’, ‘-’, ‘6’, ‘:’, ‘%’, ‘^’, ‘\n’, ‘x’, ‘~’ … Saturates after many
(~10000) iterations In less than 1 second COVER_ME is not
reached
Example ( ) {1: state = 0;2: while(1) {3: s = input();4: c = input();5: if(c==‘:’ && state==0) state=1;6: else if(c==‘\n’ && state==1) state=2;7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) {
COVER_ME:; } }}
Example ( ) {1: state = 0;2: while(1) {3: s = input();4: c = input();5: if(c==‘:’ && state==0) state=1;6: else if(c==‘\n’ && state==1) state=2;7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) {
COVER_ME:; } }}
![Page 41: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/41.jpg)
Hybrid Concolic Testing Random Phase
‘$’, ‘&’, ‘-’, ‘6’, ‘:’, ‘%’, ‘^’, ‘\n’, ‘x’, ‘~’ … Saturates after many
(~10000) iterations In less than 1 second COVER_ME is not
reached
Concolic Phase s[0]=‘I’, s[1]=‘C’,
s[2]=‘S’, s[3]=‘E’ Reaches COVER_ME
Example ( ) {1: state = 0;2: while(1) {3: s = input();4: c = input();5: if(c==‘:’ && state==0) state=1;6: else if(c==‘\n’ && state==1) state=2;7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) {
COVER_ME:; } }}
Example ( ) {1: state = 0;2: while(1) {3: s = input();4: c = input();5: if(c==‘:’ && state==0) state=1;6: else if(c==‘\n’ && state==1) state=2;7: else if (s[0]==‘I’ && s[1]==‘C’ && s[2]==‘S’ && s[3]==‘E’ && state==2) {
COVER_ME:; } }}
![Page 42: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/42.jpg)
Hybrid Concolic Testing 4x more coverage than random testing 2x more coverage than concolic testing
![Page 43: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/43.jpg)
Results
![Page 44: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/44.jpg)
Results: Red Black Treetest_driver()
RedBlackTree rb = new RedBlackTree();while(1) {
choice = input();data = input();switch(choice) {
case 1: rb.add(data); break;case 2: rb.remove(data); break;case 3: rb.find(data); break;default: rb.add_if_not_member(data); break;
}}
}
![Page 45: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/45.jpg)
Results
![Page 46: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/46.jpg)
Summary
Concolic Testing
Random Testing
![Page 47: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/47.jpg)
Summary
Concolic Testing
Random Testing
Hybrid Concolic Testing
![Page 48: Hybrid Concolic Testing](https://reader033.vdocument.in/reader033/viewer/2022051820/56814a2e550346895db74e84/html5/thumbnails/48.jpg)
Thank You!