software testing and reliability
DESCRIPTION
Software Testing and Reliability. Southern Methodist University CSE 7314. Functional and structural approaches to testing. A few words on combinatorics. Based on the Cartesian product of sets, we can count the number of possible inputs that a program has, i.e. | I |. Example. - PowerPoint PPT PresentationTRANSCRIPT
Software Testing
and Reliability
Southern Methodist UniversityCSE 7314
Functional and structural approaches to testing
A few words on combinatorics
• Based on the Cartesian product of sets, we can count the number of possible inputs that a program has, i.e. | I |
Example • Assume a program has a single
input, Customer ID (CID)• May be any value in the domain
{00000-99999}• What is | I | ?
Example• Now assume we add a second
input, the Order ID (OID)• This may be any value in the
domain {00000-99999} as well• Now what is | I |?
Example• Finally, add a credit card number to
the input• This is a 12 digit number• | I | has now reached 10**22
• If we can execute 1 million tests per second, it will take 1016 seconds, or about 300 million years!!
Example• Since we cannot know what data
may exercise a given statement/path in general, we may attempt to resort to exhaustive testing• This attempt is doomed to fail
due to the combinatorial explosion
Engineering the testing process
• Any engineered product (and most other things) can be tested in one of two ways–Knowing the specified function that a
product has been designed to perform, tests can be conducted that demonstrate each function is fully operational while at the same time searching for errors in each function
Engineering the testing process
–Knowing the internal workings of a product, tests can be conducted to ensure that “all gears mesh” (internal operations performed according to specifications)
Structural testing• Uses knowledge of the internal
workings• Also known as Clear box/glass box• Code based• Can be useful for finding interesting
inputs• Misses an entire class of faults,
missing code
Behavioral • Uses knowledge of the specific
function that is to be performed• Based solely on the specification
without regard for the internals• Also known as Black box• More user oriented• Misses an entire class of faults, extra
code (surprises) except by accident
Passing criteria• How do we know when• 1. a single test has passed• 2. when we are done testing
Passing criteria• A single test passes when its
output is correct–This requires a specific definition
of correct and ties into the automated oracle problem
When are we done?• Conway Criteria:• No syntactic errors (it compiles)• No compile errors or immediate
execution failures• There exists Some set of data for which
the program gives the correct output• A typical set of data produces the
correct output
When are we done?• Difficult sets of data produce the
correct output.• All possible data sets in the problem
specification produce the correct output
• All possible data sets and likely erroneous input succeeds.
• All inputs produce the correct output
Nature of software defects
• Logic errors and incorrect assumptions are inversely proportional to the probability that a program path will be executed
Nature of software defects
• We often believe that a logical path is not likely to be executed when, in fact, it may be executed on a regular basis
• Typographical errors are random
More of a case for WHITE box testing……
Overview of Coverage testing
Logical coverage• What does statement coverage
imply ?
Example • 1 IF (( X > 1) AND ( Y == 1))• 2 Z = Z / X• 3 END• 4 IF ((X == 2) OR ( Z >1))• 5 Z = Z + 1• 6 END
What is required for statement coverage ?
Example • To achieve statement coverage, one
can choose x = __, y = __
• Many possible errors are missed:
• 1 IF ((X > 1) OR (Y ==1))• 1 IF ((X >=1) AND (Y ==1))• 1 IF ((X > 1) AND (Y >=1))
Blindness • Statement coverage is blind to
several classes of error when choosing values to achieve coverage
Assignment blindness• Due to assignment of a particular
value to a variable, the error does not propagate
Equality blindness• Due to an Equality check of a
particular variable, the error does not propagate
Self blindness• Conditional itself covers up the
error
Assignment Blindness example
• 1 IF (<conditional>)• 2 X = 1• 3 END• 4 IF (X + Y > 1) //Should have
been (Y > 0)• 5 ....• 6 END
Assignment Blindness example
• In this example, values are chosen to make the conditional true, the statement X=1 is executed and the error in line 4 is not seen
Assignment Blindness example
• If every path forces the same assignment, then the 'error' doesn't really matter, (does it exist??) –For instance, if the conditional in
statement 1 always evaluated to true
Equality Blindness Example
• 1 IF (A == 2)• 2 .....• 3 IF (B > 1) // Should have been
(A + B > 3)
Equality Blindness Example
• In this example, the value of 2 is chosen for A to force execution of the body
• The error in statement 3 is missed
Self Blindness• 1 IF (X < 1) // Should have been
(2X < 1)
• In this example, the value of 0 is chosen for X
Observation • Statement coverage is the
weakest form of coverage• Many classes of errors can
escape this testing• Typical projects have less than
80% statement coverage!!
Branch coverage• Each branch of a decision must be
executed at least once• This is stronger that statement
coverage, but it is still weak
• Note: A decision is a logical combination of one or more conditions
Branch coverage example • 1 IF (( X > 1) AND ( Y == 1))• 2 Z = Z / X• 3 END• 4 IF ((X == 2) OR ( Z >1))• 5 Z = Z + 1• 6 END
Branch coverage example• To achieve branch coverage, one
option is to choose–x = __, y = __–x = __, y = __
Branch coverage example• Many possible errors are still
missed:
• 1 IF ((X > 1) OR ( Y ==1)• 1 IF ((X >=1) AND (Y ==1))• 1 IF ((X > 1) AND (Y <=1))
Blindness in branch coverage
• Branch coverage is blind to several classes of error when choosing values to achieve coverage–Compound decisions (more than
one condition) are weakly tested–Boundaries of conditions (within a
decision) are not explicitly tested
Condition coverage• Each condition within a decision
must assume all possible values• This is 'potentially' stronger than
branch coverage, but not always• It may, in fact, be weaker
Example of condition coverage
• 1 IF (( X > 1) AND ( Y == 1))• 2 Z = Z / X• 3 END• 4 IF ((X == 2) OR ( Z >1))• 5 Z = Z + 1• 6 END
Example of condition coverage
• To achieve condition coverage for statement 1, one must choose values such that:
• X > 1 ~(Y == 1)• ~(X > 1) Y == 1
Example of condition coverage
• Both of these vectors miss the execution of statement 2
• A better choice may be:
• ~(X > 1) ~(Y == 1)• X > 1 Y == 1
Multi-condition coverage• Every combination of possible
condition values within a decision must be chosen
• This is strictly stronger than branch or condition coverage–still has weaknesses!
Multi-condition coverage example
• 1 IF (( X > 1) AND ( Y == 1))• 2 Z = Z / X• 3 END• 4 IF ((X == 2) OR ( Z >1))• 5 Z = Z + 1• 6 END
Multi-condition coverage example
• To achieve multi-condition coverage for statement 1, one must choose values such that:
• 1. X > 1 Y == 1• 2. X > 1 ~(Y == 1)• 3. ~(X > 1) Y == 1• 4. ~(X > 1) ~(Y == 1)
Multi-condition coverage example
• To achieve multi-condition coverage for statement 2, one must choose values such that:
• 5. X == 2 Z > 1• 6. X == 2 ~(Z > 1)• 7. ~(X == 2) Z > 1• 8. ~(X == 2) ~(Z > 1)
Coverage X Y Z Covers
1 0 1 1,5
2 1 4 4,8
1 1 3 2,6
2 0 1 3,7
Multi-condition coverage example
• There are 9 paths through this code and this test set only covers 4 of those
Multi-condition coverage example
• Consider this error• Line 2 is incorrectly written as Z
= Z –X• Only the second vector will
execute this line but for the values chosen, Z-X = Z/X
Multi-condition coverage example
• Also consider that no vector will execute both line 2 and line 5
• In a more complex example, it is easy to imagine variables being set in line 2 that are then incorrectly used in line 5
Multi-condition coverage example
• For instance:
• 2. Z = Z/X; A = 0• ....• 5. Z = Z + 1; B = 1/A;
Path coverage• Every possible path must be
chosen
• Finding vectors to accomplish this is, in general, undecidable, and usually difficult
Path coverage• This still misses boundary
conditions• If statement 1 was mis-typed as
X >= 1, it is conceivable that the chosen test set does not contain a vector with X=1, missing this error
Basis path testing
Basis path testing• White box technique• Test case designer can derive a
logical complexity measure of a procedural design
• Control flow represented by a control graph
Flow graph notation
sequence
if
Flow graph notationwhile
while
Flow graph notation
case
Flowchart 1
2
3
4
5
6
7 8
910
11
Flow graph1
2,3
6
7 8
9
10
11
4,5
R3
R2
R1
R4region
nodeedge
Compound logicPredicate nodes a
b
y x
x
IF a OR bthen procedure xelse procedure y
ENDIF
Cyclomatic complexity• Software metric that provides a
quantitative measure of the logical complexity of a program
• Value computed defined the number of independent paths in the basis set of a program
• Upper bound for the number of tests that must be conducted to ensure all statements have been executed at least once
Independent path• Any path through the program
that introduces at least one new set of processing statements or a new condition
• Must move along at least one edge that has not been traversed before the path is defined
Example • Path 1; 1-11• Path 2; 1-2-3-4-5-10-1-11• Path 3; 1-2-3-6-8-9-10-1-11• Path 4; 1-2-3-6-7-9-10-1-11• 1-2-3-4-5-10-1-2-3-6-8-9-10-1-11 is
not independent because no new edges traversed
Basis set• Paths 1,2,3, and 4 constitute the
basis set• If tests can be designed to force
execution of these paths, every statement in the program will have been guaranteed to be executed at least one time and every condition will have been executed on its true and false sides
Basis set• Basis set is not unique; a
number of different basis sets can be derived for a given procedural design
How do we know how many paths to look for?
• Compute the cyclomatic complexity
• A graph theoretic computation which yields a good software metric
Three ways of computing• 1. The number of regions of the flow
graph correspond to the cyclomatic complexity
• 2. V(G) = E – N + 2– E = number of flow graph edges– N = number of flow graph nodes
• 3. V(G) = P + 1– P = number of predicate nodes in G
V(G) for Flow graph1
2,3
6
7 8
9
10
11
4,5
R3
R2
R1
R4region
nodeedge
Example • 1. Flow graph has 4 regions• 2. V(G) = 11 edges – 9 nodes + 2 = 4• 3. V(G) = 3 predicate nodes + 1 = 4• V(G) provides upper bound on
number of tests that must be designed and executed to guarantee coverage of all program statements
Deriving test cases• Basis path testing can be applied
to a procedural design or to source code
PDL for test case designPROCEDURE average;* This procedure computes the average of 100 or fewer numbers that lie between bounding values; it also computes the sum and the total number valid.INTERFACE RETURNS average, total.input, total.valid;INTERFACE ACCEPTS value, minimum, maximum;
TYPE value[1:100] IS SCALAR ARRAY;TYPE average, total.input, total.valid; minimum, maximum, sum IS SCALAR;TYPE i IS INTEGER
i = 1;total.input = total.valid = 0;sum =0;DO WHILE value[i] <> -999 AND total.input < 100; increment total.input by 1; IF value[i] >= minimum AND value[i] <= maximum THEN increment total.valid by 1; sum = sum + value[i]; ELSE skip ENDIF increment i by 1;ENDDOIF total.valid > 0 THEN average = sum / total.valid; ELSE average = -999ENDIFEND average
Deriving test cases for “average”
• 1. Using the design or code as a foundation, draw a corresponding flow graph
i = 1;total.input = total.valid = 0;sum =0;DO WHILE value[i] <> -999 AND total.input < 100; increment total.input by 1; IF value[i] >= minimum AND value[i] <= maximum THEN increment total.valid by 1; sum = sum + value[i]; ELSE skip ENDIF increment i by 1;ENDDOIF total.valid > 0 THEN average = sum / total.valid; ELSE average = -999ENDIFEND average
Identify the nodes in this PDL!!!
Draw the Flow graph
Deriving test cases for “average”
• 2. Determine the cyclomatic complexity of the resultant flow graph–V(G) can be determined without
developing a flow graph by counting all conditional statements in the PDL (compound conditions count as two) and then adding 1
Computer V(G) using the three methods
• 2. (continued)–V(G) = –V(G) =–V(G) =
Deriving test cases for “average”
• 3. Determine a basis set of linearly independent paths– Value of V(G) provides the number of
linearly independent paths• Path 1; • Path 2; • Path 3; • Path 4; • Path 5; • Path 6;
List the basis set paths!!
Deriving test cases for “average”
• 4. Prepare test cases that will force execution of each path in the basis set–Data should be chosen to that
conditions at the predicate nodes are appropriately set as each path is tested. Test cases that satisfy the basis set just described are;
Prepare a set of test cases
Summary • Each test case is executed and
compared to expected results• Ensures all statements executed
at least once• Some independent paths cannot
be tested in stand-alone fashion–Tested as part of another path test
Graph matrices
Graph matrices• Testing basis paths can be
mechanized by using graph matrices
• Size based on number of nodes in the graph
• Tabular representation of a flow graph
Graph matrix
1
3
4
2
5
a
b
cd
e
fg
Graph matrix
a
d b
c f
g e
1 2 3 4 5
1
2
3
4
5
node
Connected to node
Link weight• Provides additional information
about control flow• Connection either exists (1) or does
not exist (0)• Can also represent– Probabilities of execution– Processing time of a link–Memory or resources required
Connection matrix
1
1 1
1 1
1 1
1 2 3 4 5
1
2
3
4
5
node
Connected to node
connections
1 – 1 = 0
2 – 1 = 1
2 – 1 = 1
2 – 1 = 1
3 + 1 = 4Cyclomatic complexity
Data flow testing
Data flow testing• This approach selects test paths
according to the locations of definitions and uses of variables in the program
• Definition Use (DU) testing–Does not guarantee coverage of all
branches of a program but does pretty good
Example proc x B1; do while C1 if C2 then
if C4 then B4; else B5;endif;
else if C3
then B2; else B3;endif;
endif; enddo; B6;end proc;
Now apply the DU testing strategy
• We must first know the definitions and uses of variables in each condition or block in the PDL
• Assume variable x is defined in the last statement of blocks B1, B2, B3, B4, and B5 and used in the first statement of blocks B2, B3, B4, B5, and B6
Now apply the DU testing strategy
• The DU testing strategy requires an execution of the shortest path from each of Bi, 0<i<5, to each of Bj, 1<j≤6
• This testing also covers any use of variable x in conditions C1, C2, C3, and C4
Now apply the DU testing strategy
• There are 25 DU chains of variable x, but we need only five paths to cover these DU chains
• 5 paths are needed to cover the DU chain of x from Bj, 1<j≤5, and other BU chains can be covered by making these five paths contain iterations of the loop
Loop testing
Loop testing• Loops are the cornerstone of many
algorithms in software• White box testing technique• Four different classes of loops– Simple loops– Concatenated loops– Nested loops– Unstructured loops
Simple loops
Testing Simple loops• Skip the loop entirely• Only one pass through the loop• Two passes through the loop• m passes through the loop where m
< n (n is maximum allowable passes through the loop)
• n-1, n, n+1 passes through the loop
Nested loops
Nested loops• Cannot simply extend the
approach from simple loops–Can grow geometrically
• 1. Start at the innermost loop. Set all other loops to minimum values
Nested loops• 2. Conduct simple loop tests for
the innermost loop while holding the outer loops at their minimum iteration parameter (e.g. loop counter) values. Add other tests for out of range or excluded values
Nested loops• 3. work outward, conducting
tests for the next loop, but keeping all other outer loops at minimum values and other nested loops to “typical” values
• 4. continue until all loops have been tested
Concatenated loops
Concatenated loops• These loops can be tested using the
approach defined for simple loops, if each of the loops is independent of the other
• However, if two loops are concatenated and the loop counter for loop 1 us used as the initial value for loop 2, then the loops are not independent. When this happens, use the nesting loops approach
Unstructured loops
Unstructured loop• Redesign !!!• Use structured programming
constructs• http://acweb.colum.edu/users/
rcourington/Progclass/struprog.html
End of lecture