t.y. chen swinburne university of technology, australia
DESCRIPTION
Semi-Proving : an Integrated Method Based on Global Symbolic Evaluation and Metamorphic Testing. T.Y. Chen Swinburne University of Technology, Australia. T.H. Tse and Zhiquan Zhou The University of Hong Kong. ( speaker). Presentation Outline. Conventional Program Testing and Proving - PowerPoint PPT PresentationTRANSCRIPT
1
T.Y. Chen
Swinburne University of Technology, Australia
T.H. Tse and Zhiquan Zhou
The University of Hong Kong
Semi-Proving: an Integrated Method Based on Global Symbolic Evaluation and Metamorphic Testing
(speaker)
2
Presentation Outline
Conventional Program Testing and Proving Metamorphic Testing Our method: Semi-Proving Summary.
3
Conventional Program Testing and Proving Metamorphic Testing Our method: Semi-Proving Summary.
Presentation Outline
4
Conventional Program Testing and Proving
Given a bijective function f ;
A Program: F_Sort (a1, a2, ..., an), n 2
Output: (a1’, a2’, ..., an’), such that
1. (a1’, a2’, ..., an’) is a permutation of (a1, a2, ..., an)
2. f (a1’) f (a2’) ... f (an’).
Given a bijective function f ;
A Program: F_Sort (a1, a2, ..., an), n 2
Output: (a1’, a2’, ..., an’), such that
1. (a1’, a2’, ..., an’) is a permutation of (a1, a2, ..., an)
2. f (a1’) f (a2’) ... f (an’).
5
Conventional Program Testing and Proving
Testing
1. Design test cases: e.g. (2, 6, 3) for n=3
2. Run: F_Sort (2, 6, 3) = (6, 3, 2)
3. Check: f (6) < f (3) < f (2) ?
1. Design test cases: e.g. (2, 6, 3) for n=3
2. Run: F_Sort (2, 6, 3) = (6, 3, 2)
3. Check: f (6) < f (3) < f (2) ?
6
Conventional Program Testing and Proving
Proving correctness
1. F_Sort terminates for any valid input;
2. The output is correct.
1. F_Sort terminates for any valid input;
2. The output is correct.
7
Conventional Program Testing and Proving
Proving properties
F_Sort (a1, a2, ..., an) = (a1’, a2’, ..., an’) F_Sort (a1, a2, ..., an) = (a1’, a2’, ..., an’)
Permutation.
8
Metamorphic Testing
Metamorphic Testing
Employing relationships between different executions
Fact: different permutations will produce same output
F_Sort (a1, a2, a3)
Fact: different permutations will produce same output
F_Sort (a1, a2, a3) F_Sort (a3, a1, a2) = “ Metamorphic Relation ” ·
9
Metamorphic Testing
Metamorphic Test Cases: {(2, 6, 3), (3, 2, 6)}Metamorphic Test Cases: {(2, 6, 3), (3, 2, 6)}
Metamorphic Testing:
1. F_Sort (2, 6, 3) = (6, 3, 2)
Metamorphic Testing:
1. F_Sort (2, 6, 3) = (6, 3, 2)
No matter whether an oracle is available or not;Very useful when the oracle cannot be found.
2. F_Sort (3, 2, 6) = (6, 3, 2)| |
PASS
10
Metamorphic Testing
Metamorphic Test Cases: {(2, 6, 3), (3, 2, 6)}Metamorphic Test Cases: {(2, 6, 3), (3, 2, 6)}
Metamorphic Testing: 1. F_Sort (2, 6, 3) = (6, 3, 2)
Metamorphic Testing: 1. F_Sort (2, 6, 3) = (6, 3, 2)
2. F_Sort (3, 2, 6) = (3, 6, 2) Failure.| |
11
Conventional Program Testing and Proving Metamorphic Testing Semi-Proving: Verifying Metamorphic
Relations Summary.
Presentation Outline
12
Semi-Proving: Verifying Metamorphic Relations
Objective:
If the program does not satisfy a metamorphic relation on some inputs, locate these inputs;
Otherwise prove the satisfaction of the metamorphic relation over all inputs.
13
Why called “Semi”?
Proving necessary properties, which may not be sufficient for program correctness
Characteristics of Semi-Proving
Multiple symbolic executions
Testing and proving.
Semi-Proving: Verifying Metamorphic Relations
14
double GetMid (double x1, double x2, double x3){ double mid;
mid = x3;if (x2 < x3)
if (x1 < x2)mid = x2;
else {if (x1 < x3)
mid = x1;}
elseif (x1 > x2)
mid = x2;else if (x1 > x3)
mid = x1; return mid;
}
double GetMid (double x1, double x2, double x3){ double mid;
mid = x3;if (x2 < x3)
if (x1 < x2)mid = x2;
else {if (x1 < x3)
mid = x1;}
elseif (x1 > x2)
mid = x2;else if (x1 > x3)
mid = x1; return mid;
}
Semi-Proving: Verifying Metamorphic Relations
15
Specification
“GetMid (X, Y, Z)” returns the median of (X, Y, Z)
E.g. GetMid (3, 4, 1): “3”.
Semi-Proving: Verifying Metamorphic Relations
16
Verifying “GetMid” by Semi-Proving
Identify a Metamorphic Relation
GetMid ( X, Y, Z ) = GetMid ( permute(X, Y, Z) )
Semi-Proving: Verifying Metamorphic Relations
any numbers any permutation
Purpose: to verify
17
Basic concepts
Transposition
• simple permutation that exchanges two elements
(1, 2, 3)
(1, 2, 3)
......... 1
(1, 2, 3) (1, 3, 2) ......... 2
(2, 1, 3)
Semi-Proving: Verifying Metamorphic Relations
18
A tuple (1, 2, 3)
A permutation (2, 3, 1)
(1, 2, 3)
A tuple (1, 2, 3)
A permutation (2, 3, 1)
(1, 2, 3) (2, 3, 1)1 (2, 1, 3) 2
Basic concepts
Composition of Transpositions
Semi-Proving: Verifying Metamorphic Relations
19
Result from Group Theory
Any permutation of (X, Y, Z) can be achieved by compositions of transpositions (X, Z, Y) and (
Y, X, Z).
Semi-Proving: Verifying Metamorphic Relations
20
Semi-Proving: Verifying Metamorphic Relations
Purpose
GetMid ( X, Y, Z ) = GetMid ( permute(X, Y, Z) )
Only need to verify:
Any permutation.
• GetMid (X, Y, Z) = GetMid (X, Z, Y)
• GetMid (X, Y, Z) = GetMid (Y, X, Z)
21
Semi-Proving: Verifying Metamorphic Relations
Purpose
GetMid ( X, Y, Z ) = GetMid ( permute(X, Y, Z) )
Only need to verify:
• GetMid (X, Y, Z) = GetMid (X, Z, Y)
• GetMid (X, Y, Z) = GetMid (Y, X, Z)
22
Global Symbolic Evaluation on GetMid (X, Y, Z)
Execute all the possible paths.
Semi-Proving: Verifying Metamorphic Relations
23
double GetMid (double x1, double x2, double x3){ double mid;
mid = x3;if (x2 < x3)
if (x1 < x2)mid = x2;
else {if (x1 < x3)
mid = x1;}
elseif (x1 > x2)
mid = x2;else if (x1 > x3)
mid = x1; return mid;
}
double GetMid (double x1, double x2, double x3){ double mid;
mid = x3;if (x2 < x3)
if (x1 < x2)mid = x2;
else {if (x1 < x3)
mid = x1;}
elseif (x1 > x2)
mid = x2;else if (x1 > x3)
mid = x1; return mid;
}
Semi-Proving: Verifying Metamorphic Relations
24
C1: (Y X < Z) OR (Z < X Y)
Path Conditions C2: (X < Y < Z) OR (Z Y < X)
C3: (Y < Z X) OR (X Z Y)
Semi-Proving: Verifying Metamorphic Relations
X when C1 is trueGetMid (X, Y, Z) = Y when C2 is true
Z when C3 is true
25
Semi-Proving: Verifying Metamorphic Relations
?GetMid (X, Z, Y)
?X when C1 is true
GetMid (X, Y, Z) = Y when C2 is trueZ when C3 is true
26
C4: (Z X < Y) OR (Y < X Z)
C5: (X < Z < Y) OR (Y Z < X)
C6: (Z < Y X) OR (X Y Z)
PASS
Semi-Proving: Verifying Metamorphic Relations
?GetMid (X, Z, Y)
?X when C4 is true
= Z when C5 is true
Y when C6 is true
X when C1 is trueGetMid (X, Y, Z) = Y when C2 is true
Z when C3 is true
27
? ?
X when C1 is trueGetMid (X, Y, Z) = Y when C2 is true
Z when C3 is true
X when C4 is true
= Z when C5 is true
Y when C6 is true
Contradiction
PASS
C1: (Y X < Z) OR (Z < X Y) &
Semi-Proving: Verifying Metamorphic Relations
GetMid (X, Z, Y)?
C4: (Z X < Y) OR (Y < X Z)
C5: (X < Z < Y) OR (Y Z < X)
C6: (Z < Y X) OR (X Y Z)
28
? ?
C4: (Z X < Y) OR (Y < X Z)
C5: (X < Z < Y) OR (Y Z < X)
C6: (Z < Y X) OR (X Y Z)
X when C1 is trueGetMid (X, Y, Z) = Y when C2 is true
Z when C3 is true
X when C4 is true
= Z when C5 is true
Y when C6 is true
C1: (Y <= X < Z) OR (Z < X <= Y) &
X=Y<Z OR Z<Y=X
Semi-Proving: Verifying Metamorphic Relations
?GetMid (X, Z, Y)
29
? ? ?
C4: (Z X < Y) OR (Y < X Z)
C5: (X < Z < Y) OR (Y Z < X)
C6: (Z < Y X) OR (X Y Z)
X when C1 is trueGetMid (X, Y, Z) = Y when C2 is true
Z when C3 is true
X when C4 is true
= Z when C5 is true
Y when C6 is true
C1: (Y <= X < Z) OR (Z < X <= Y) &
Yes. X=Y
PASS
X=Y<Z OR Z<Y=X
Semi-Proving: Verifying Metamorphic Relations
GetMid (X, Z, Y)
30
?
X when C1 is trueGetMid (X, Y, Z) = Y when C2 is true
Z when C3 is true
Semi-Proving: Verifying Metamorphic Relations
GetMid (X, Z, Y)
verified
31
?
X when C1 is trueGetMid (X, Y, Z) = Y when C2 is true
Z when C3 is true
Semi-Proving: Verifying Metamorphic Relations
Conclusion
GetMid (X, Z, Y)
32
?
X when C1 is trueGetMid (X, Y, Z) = Y when C2 is true
Z when C3 is true
Semi-Proving: Verifying Metamorphic Relations
Conclusion
GetMid (X, Z, Y)
33
X when C1 is trueGetMid (X, Y, Z) = Y when C2 is true
Z when C3 is true
Semi-Proving: Verifying Metamorphic Relations
Conclusion
GetMid (X, Z, Y)
34
X when C1 is trueGetMid (X, Y, Z) = Y when C2 is true
Z when C3 is true
Semi-Proving: Verifying Metamorphic Relations
Conclusion
GetMid (X, Z, Y)
Composition of transpositions
GetMid (X, Y, Z) = GetMid ( Permute(X, Y, Z) )
GetMid (Y, X, Z)
Any Any.
35
Detecting Program Faults ·
Semi-Proving: Detecting Program Faults
36
double GetMid (double x1, double x2, double x3){ double mid;
mid = x3;if (x2 < x3)
if (x1 < x2)mid = x2;
else {if (x1 < x3)
mid = x1;}
elseif (x1 > x2)
mid = x2;else if (x1 > x3)
mid = x1; return mid;
}
double GetMid (double x1, double x2, double x3){ double mid;
mid = x3;if (x2 < x3)
if (x1 < x2)mid = x2;
else {if (x1 < x3)
mid = x1;}
elseif (x1 > x2)
mid = x2;else if (x1 > x3)
mid = x1; return mid;
}
37
Verify: GetMid (X, Y, Z) = GetMid (X, Z, Y)
Semi-Proving: Detecting Program Faults
| |
X
when Y X < Z
?| |
Y
when (Z < Y X ) OR (Y Z AND X Z) AND
38
Verify: GetMid (X, Y, Z) = GetMid (X, Z, Y)
Semi-Proving: Detecting Program Faults
| |
X
when Y X < Z
?| |
Y
when (Z < Y X ) OR (Y Z AND X Z) AND
(Y=X<Z) OR (Y<X<Z)
39
Verify: GetMid (X, Y, Z) = GetMid (X, Z, Y)
Semi-Proving: Detecting Program Faults
| |
X
when Y X < Z
?| |
Y
when (Z < Y X ) OR (Y Z AND X Z) AND
(Y=X<Z) OR (Y<X<Z)
?
failure
Failure-causing inputCan identify all the
failure-causing inputs.
40
Summary A proving technique: all the paths A testing technique:
failure-causing inputs selected path(s)
Characteristics Metamorphic relations Multiple symbolic executions Employing global symbolic evaluation and constraint
solving.
41
Questions are welcome