eecs814: sep 8,04 - university of kansas · web viewnot improve process not better product not...
TRANSCRIPT
Introduction
Product Development:- Ineffective: organization focus on cost.- Effective: make the process efficient
Define Quality:- is conformance to request.- is fitness for intended users.- …
Definitions of Software Quality:- IEEE:-
Summary of Defining Quality::- …- …
The Evolution of Quality:1. Past: Quality Control2. Now: Quality Assurance3. Future: Quality Engineering
(Past)Quality Control
(Now)Quality Assurance
(Future)Quality Engineering
)1(Aug 25, 05
The Evolution of Quality
1. Quality Control:- Quality Control is defined as a set of procedures designed to ensure that quality standards and
processes are adhered to, and that the final product meets or exceeds the required technical and performance requirements, so it keeps quality at an acceptable level by rejecting unacceptable.
- Prevent unacceptable products from being released to the customer.- Problems:
o Not reduce defect.o Not improve processo Not better producto Not motivate improvement
2. Quality Assurance:- Covers all activities from design, development, production, installation, servicing and
documentation, this introduced the rules: "fit for purpose" and "do it right the first time".- Providing evidence that the quality function is being performed adequate.- IEEE: A planned and systematic pattern of all actions necessary to provide adequate confidence
that the product conforms to established technical requirement.- System of methods and procedures used to assure that the software product meet.- Pros:
o More effective than quality because emphasize on development processing.o Fix problems during and before development processo Improve process reduce defect in a last manner
- Coso Separate from other
process improve and software development activity.
o Motivation to improve is inconsistent
o Cost of making people monitors others.
3. Quality Engineering:- Similar to Quality Assurance, but responsibility on everyone.- Problems result in process change, not punishment of people.- Finding error is good – keeping them leak to customers.- Appreciates that a competitive process is the way to remain a competitive.- Measurement are used so decisions based on face (in addition to institution)- Pride in quality- Build quality in product through development- Less adversarial- Motivation- Flexible to change the process in response to a problem
o Understand the problem & cause & process- Knowledge is the foundation of successfully quality engineering.
)2(
Development PC Inspection PassRequest
Fail
Development PC Inspection PassRequest
Fail
Process & DesignStandards
Standard of Quality
A Perspective on Testing
Definition:The process of executing a program with the intend of finding errors.
Testing Objective:- To improve quality by finding error.- To provide confidence in the dependability of software.
Related Terms:- Error: A measure of the difference
between the actual and the ideal. It occurs in some hardware or software component.
- Fault: A condition that causes a system to fail in performing its required function. Fault is a manifestation of an error.
- Failure: The inability of a system or component to perform a required function according to its specifications. Failure is a departure of a system from the service required.
Static vs. Dynamic:- Static Analysis is analysis done on source code without actually executing it.
o Defects like dead-code (i.e., unreachable code) can only be detected through careful inspection.
o Similarly, non-conformance to organization-wide coding conventions can also be detected through inspection process only.
o Syntax errors are caught by static analysis- Dynamic Analysis is analysis done as a program is executing and is based on intermediate
values that result from the program’s execution.o E.g., A division by 0 error is caught by dynamic analysis.
Verification vs. Validation:- Verification:
o Are we building the right product?
o Does the software fulfill its requirement?
- Validation:o Are we building the
product right?o Is implementing consistent with its specification?
)3(
Fault, Failure, and Error: (Scenario)
Consider for instance a system running on a multi-processor architecture: a fault in one processor might cause it to crash (i.e., a failure), which will be seen as a fault of the system. Therefore, the ability of the system to function even in the presence of the failure of one processor will be regarded as fault-tolerance instead of failure-tolerance. (Not all faults cause immediate failure.)
SystemActual
user need
Validation Verification
A Perspective on Testing (Cont.)
If testing doesn't find any defect?- Low quality testing- High quality software- Metrics are needed- Quality assessment (what percentage of defect captured, what percentage remain).- Risk management.
Test Cases:- Valuable (as much as source program)- The information need to be in a test case:
o Input: Preconditions The actual inputs.
o Expected Output: Postconditions Actual outputs.
o Optional: ID Purpose Execution history
- Testing Oracles: A testing oracle is any program, process, or body of data that specifies the expected outcome of a set of tests. Oracles are often defined as a set of input/expected outcome pairs.
Relationship between specified, implemented, and tested behaviors:
- Region 2 and 5: There may be specified behaviors that are not tested.- Region 1 and 4: Specified behaviors that are tested.- Region 3 and 7: Test cases that correspond to unspecified behaviors.- Region 2 and 6: There may be programmed behaviors that are not tested.- Region 1 and 3: Programmed behaviors that are tested.- Region 4 and 7: Test cases that correspond to unprogrammed behaviors.- Note: We need to make the region where these sets all intersect (Region 1) as large as possible.
65
7
41
3
2
Specification(expected) Program
(observed)
Test Cases(verified)
)4(
Test Case ID
Purpose
Preconditions
Inputs
Expected Outputs
Postconditions
Execution History Date Result Version Run By
A Perspective on Testing (Cont.)
Testing strategy:- Unit Testing: A unit is the smallest testable piece of
software (e.g., function).- Component Testing: A component is an integrated
aggregate of one or more units (e.g., module).- Integration Testing: This testing is used to
demonstrate that a combination of successfully tested components has bugs.
- System Testing: A system is a very large component. System testing includes testing for:
o performanceo securityo system recovery from failure
Concept map of testing:
Testing vs. Debugging:- Debugging: The activity by which faults are identified and rectified.- The purpose of testing is to show that a program has bugs.- The purpose of debugging is to find the faults that led to the program’s failure and to design and
implement the program changes that correct the faults.- Testing is a demonstration of failure or apparent correctness.- Debugging is a deductive process.- Testing proves a programmer’s failure.- Debugging is a programmer’s vindication. - Testing can be automated to a large extent.- Automatic debugging is still a dream. - Much of testing can be done without design knowledge (by an outsider).- Debugging is impossible without detailed design knowledge (by an insider).
SW under test
Requirement
Test set
Test casegeneration
Result
Tester
Test implementation
of execution
assumption method Test purpose
?
)5(
Uni
tTe
stin
g
Com
pone
ntTe
stin
g
Inte
grat
ion
Test
ing
Sys
tem
Test
ing
Testing Techniques
1. Functional (Black Box) Testing:
- Program is treated as a black box.- Program is subjected to inputs, and its outputs are verified for conformance to specified
behavior. - Implementation details do not matter. - Takes a user’s point of view.- In principle, can detect all bugs in an infinite amount of time.- The program test cases are based on specifications.- Test planning can begin early in the software process; test case generation can be in parallel.- A complete functional test would consist of subjecting a program to all possible input streams.
2. Structural (White Box) Testing:- Aims at exercising the different control and data structures used in the program. - Criteria are precise as they are based on program structures (i.e., are quite precise). - Looks at implementation details.- Takes a developer’s point of view.- Is inherently finite but cannot detect all faults. - Derivation of test cases according to program structure. Knowledge of the program is used to
identify additional test cases.- Objective is to exercise all program statements (not all path combinations).- One should design enough tests to ensure that every path is executed at least once.
Which testing technique is better?- According to Venn diagrams, neither is sufficient.- If all specified behavior have not been implemented, structural testing will never been able to
recognize this.- If a program (pgm) implements behaviors that haven not been specified, this will never be
revealed by functional testing.
Input Output
)6(
1. Testing Techniques: Functional Testing
Single Fault Assumption vs. Multi Fault Assumption:- Single Fault Assumption: specifies that exactly one component is malfunctioning and the cause
of the problem.- Multi Faults Assumption: more than one component are causing of the problem.
Dependent Variables vs. Independent Variables:- Dependent Variables: such as month Feb 29 days depends on the year.
Physical vs. Logic:- Some techniques require logic recognition of data dependencies such as equivalence class. This
requires more thought and more judgment.- Other techniques are domain-based that have no recognition of data or logical dependencies
such as Boundary Value Testing.
Worst Case - 5n test cases - Multiple Faults - Valid Input
Robustness WC - 7n test cases - Multiple Faults - Invalid Input
Strong Normal - Multiple Faults - Valid Input
Strong Robust - Multiple Faults - Invalid Input
1.2 Equivalence Class Testing(In/Dependent variables – Physical / Logic)
1.1 Boundary Value Testing(Independent variables only – Physical Only)
1.3 Decision Table-Based Testing(Dependent variables only – Physical / Logic)
1.4 Cause & Effect
BVA - 5n test cases - Single Fault - Valid Input
Robustness - 7n test cases - Single Fault - Invalid Input
Weak Normal - Single Fault - Valid Class
Weak Robust - Single Fault - Invalid Input
Testing Techniques
1. Black Test
)7(Sep 1, 05
1.1 Functional Testing: Boundary Value Testing
1. Boundary Value Analysis (BVA):- Objective: to use the values at their:
o Minimum valueo Just above the min (min+1)o Just below the max (max-1)o Maximum value
- Rationale: errors occur more frequency at extreme values.- Boundary value can be character, ordinary value, …- Requires 4n+1 test cases.- Variables have discrete, bounded values.- This technique is more effective with independent variables while
with dependent variables is not effective as independent variables.
- Works well when a program is a function of independent var represent boundary physical quantity.
- When no bounds exist, we have to create artificial bounds, as in the triangle problem. Creating the lower bound and the maximum bound (called MAXINT in some language).
2. Robustness Testing:- An extension of BVA: include value below the minimum and above the maximum values.
o Negative Testing: how does the program handle errors in input?o May not be possible with some strongly by typed languages, such as compiler.
- In addition to the five values of variable, exceed the extremes with maximum and minimum.- Useful for exception handling to test behavior for invalid input
values.- Stress input boundary.- Acceptable response for invalid input?- Leads to exploratory testing (test hackers)- Discover hidden functionality.
3. Worst-Case Testing:- Do not consider single fault assumption.- Explore situation with > 1 variables in their extreme values.- More through; useful for critical software.- Number of test cases is 5n.
4. Robust Worst-Case Testing:- Discard single fault assumption.- Allow the input values to simultaneously approach and
exceed their boundaries.- 7n test cases.
Thought on BVA:- Does not require much thought.- Usually assumes the variables are independent. Variables relationships not considered.- Values at certain boundaries may not have a meaning.
)8(
V1
V2
Variablevalues
BVA test cases for a function of two variables
Robustness test cases (one variable)
V1
V2
Worst-Case Testing (Two variables)
Variablevalues
BVA test cases for a function of one variable
Boundary Val ue Test Cases for ( 'Triangle' Problem)
Test Cases:
- Boundary Value Analysis:
- Worst Case:
Boundary Value Test Cases for ( 'NextDate' Problem)
)9(
Test Cases:
- Worst-Case:
)10(
Leap Year:
In the Gregorian calendar, a normal year consists of 365 days. Because the actual length of a sidereal year (the time required for the Earth to revolve once about the Sun) is actually 365.25635 days, a "leap year" of 366 days is used once every four years to eliminate the error caused by three normal (but short) years.
1) Any year that is evenly divisible by 4 is a leap year: for example, 1988, 1992, and 1996 are leap years. However, there is still a small error that must be accounted for. To eliminate this error, the Gregorian calendar stipulates that
2) a year that is evenly divisible by 100 (for example, 1900) is a leap year only if it is also evenly divisible by 400. For this reason, the following years ARE NOT leap years 1700, 1800, 1900, 2100, 2200, 2300, 2500, 2600 because they are evenly divisible by 100 but NOT by 400. The following years ARE leap years 1600, 2000, 2400 because they are evenly divisible by both 100 and 400.
Boundary Value Test Cases for ( 'Commission' Problem)
From one corner:
Test Cases:
- Output BVA:
)11(
1.2 Functional Testing: Equivalence Class Partitioning
Introduction:- Two motivations:
o A sense of complete testing.o Avoid redundancies.
- Several variation:o Robust: consider invalid input.o Weak: single fault assumption.
- Partition a set of values into a collection of mutually disjoint subsets (union is the entire set).o Completenesso Disjointedness ensures a form of non-redundancy.
- Testing idea: identify test cases by using one element from each set.- An Example: In triangle problem: {<1}, {1 … 200}, and {>200}- Key idea in EC Testing:
o Wise selection of equivalent classes.o Assuming f(X1, X2) where:
a≤X1≤d, with intervals [a,b), [b,c), [c,d]c≤X2≤g, with intervals [e,f), [f,g]
Invalid values of X1, X2 are: X1< a, X2>d, X2<e, X2>g
Equivalence Class (EC) Partitioning:- The sets in the partitions of the domain (input, output).
o Determine the boundarieso Determine the equivalences.
- Motivation:o A sense of completenesso Minimize redundancies
- If a given set of values all cause a program to behave in exactly the same way, then that is a candidate for an EC if a test case in one EC detects on defect, all other test cases will detect the same defect.
- Input domain test cases focus on input, it may does not provide confidence about calculation.
Some comments on EC Testing:- The weak forms are not very comprehensive.- If error condition are a high priority, robust form should be considered- EC is strengthened with a hybrid approach with BVA- Strong EC presumes the variables are independent. When dependence occurs DT is appropriate.- Several tries may be needed to choose the right ECs.
)12(Sep 8, 05
Equivalence Class Test Cases for ( 'Mortgage Company' Problem)
Problem Description:- Writes mortgage for people with monthly income of $1,000/month to $30,000/month.- Writes a single mortgage for 1-4 homes- Makes mortgages on condos, town houses, singled dwelling (not on duplex, mobile home).- Will make mortgage only for a person (not to corporations, trust, partnership)
Conditions:- If an input represents a continuous range of values → Select one valid & two invalid classes.
- If an input takes on discrete values (from a possible set) → Select one valid & two invalid classes.
- If input conditions is specified as a set of valid values → Select one valid EC (continue all valid members) & one invalid EC.
- If an input specifies a "must be" condition → Select one invalid EC (representing the "must be" condition) & one invalid class that does not include the "must be" condition.
Example of Test Cases:
Income $1,000 30,000
valid invalidinvalid
# of houses6
valid invalidinvalid
543210-1
InvalidDuplex
ValidSingle mortgage
Town housesCondo
ValidPerson
InvalidCorporationsPartnership
Trust
Monthly Income
# of Dwelling
Applicant Dwelling Type Output Result
$5,000$100$5,500
289
PersonPartnershipCorporation
CondoDuplexTown home
OKnot OKnot OK
)13(
Equivalence Class Test Cases for ( 'Triangle' Problem)
The Output Equivalence Classes:
Test Cases:
- Weak Normal EC:
- Weak Robust EC:
- Strong Robust EC (from one corner of the cube:
)14(
Equivalence Class Test C ases for ( 'NextDay' Problem) – Version 1
The Input Equivalence Classes:
- Valid Classes:
- Invalid Classes:
Test Cases:
- Weak Normal EC & Strong Normal EC:
- Weak Robust EC:
- Strong Robust EC (from one corner of the cube:
)15(
Equivalence Class Tes t Cases for ( 'NextDay' Problem) – Version 2
More careful selection of ECs:- Reduce granularity: focus on more specific treatment of variables; How? Consider input data:
o If not last day of month, next date increments day.o If at the end of a month, day is set to 1, month is incremented.o If at the end of a year, month, and day are set to 1, year is incremented.o Leap year? When is the last day of a month.
The Output Equivalence Classes:
Test Cases:- Weak Normal EC:
- Strong Normal EC (from one corner of the cube:
)16(
Equivalence Class Test Cases for ( Commission' Problem) – Version 1
The Input Equivalence Classes:
- Valid Classes:
- Invalid Classes:
Test Cases:
- Strong Robust EC:
)17(
Equivalence Class Test Cases for ( Commission' Problem) – Version 2
The Output Equivalence Classes:
Output Range EC Test Cases:
)18(
1.3 Functional Testing: Decision Tables
Introduction:- Useful for capturing and presenting complex logical decision making.- Prototype:
Rules(IF) Conditions(THEN) Actions
- Example:R1 R2 R3 R4 R5 R6 R7 R8
Valid Size:Valid quantitySufficient Funds
FFF
FFT
FTF
FTF
TFF
TFT
TTF
TTT
Buy No No No No No No No Yes
- It can have multiple actions/conditions/rules.- Well suited when logical relationship exists between variables.- Choice of conditions greatly expands the table (look at the Triangle Problem below).- Number of the rules:
o With n conditions imply 2n rules.o If action sets of two rules are identical, there must be at least one condition that allows
two rules to be combined with a "-" entry. Look at the NextDay example: Rules 1-3 involve D1, D2, D3 for 30-day months → Combine! Rules 6-9 involve D1, D2, D3, D4 for 31-day months → Combine!
- In testing cases:o Conditions → Inputo Action → Outputo Rules → Test cases
Summary of Decision Tables:- Appropriate for:
o Prominent if-then-elseo Logical relationship among variableso Cause and effect relationso High cycloramic complexity
- Do not scale up well (2n combinations)o Algebraically factor large tableso Look for repeating patterns
- The combination will not work with Commission Problem because there is no relationship between the input variables (Independent variables).
)19(Sep 15, 05
Decision Tables Test Cases for ( 'Triangle' Problem)
The Rules:c1: a,b,c form a triangle (NEEDS to be expanded)
c1: a<b+c?c2: b<a+c?c3: c<a+b?
c4: a=b?c5: a=c?c6: b=c?a1: not a trianglea2: scalenea3: isoscelesa4: equilaterala5: impossible
Decision Table:
R1 R2 R3 R4 R5 R6 R7 R8 R9 R10 R11C1C2C3C4C5C6
F-----
TF----
TTF---
TTTTTT
TTTTTF
TTTTFT
TTTTFF
TTTFTT
TTTFTF
TTTFFT
TTTFFF
A1A2A3A4A5
x x x
xx x
x
x
x xx
)20(
Decision Tables Test Cases for ( 'NextDay' Problem) – Version 1
The Rules:M1 = {mon | mon has 30 days}M2 = {mon | mon has 31 days}M3 = {mon | mon is Feb}D1 = {day | 1 <= day <= 28}D2 = {day | day = 29}D3 = {day | day = 30}D4 = {day | day = 31}Y1 = {year | year = 2000}Y2 = {year | a leap year}Y3 = {year | a common year}
Decision Table:
)21(
Problem: Rule 8 has unknown decision
Decision Tables Test Cases for ( 'NextDay' Problem) – Version 2
The Rules:M1 = {mon | mon has 30 days}M2 = {mon | mon has 31 days; not Dec}M3 = {mon | mon is Dec}M4 = {mon | mon is Feb}D1 = {day | 1 <= day <= 27}D2 = {day | day = 28}D3 = {day | day = 29}D4 = {day | day = 30}D5 = {day | day = 31}Y1 = {year | year is a leap year}Y2 = {year | year is a common year}
Decision Table:
Reduced Decision Table:
Test Cases:
)22(
Retrospective of Functional Tests
Effort:- Domain-based techniques have no recognition of data or logical dependencies- The equivalent class techniques pay attention to data dependencies and to the function itself- The decision table techniques requires consider both data and logical dependencies- Trade-off between test identification effort and test execution effort
Efficiency: - The intuitive notion is that a set of test cases is just right (no gap , no redundancy)- The gap is the possibility of untested functionality.
Effectiveness:- Mandate a method, use it to generate test cases, and then run the test cases- Know how effective a set of test cases is for finding faults present in a program
Appropriate Choices for Functional Testing:- Look at the following table to see the appropriate testing techniques.
o Only Strong EC and Decision table can handle dependent variables.o Decision table works only with dependent variables.o Boundary tests work with physical variables not logical.
)23(Sep 22, 05
A Brief Review of Graphs
Introduction:- Directed graph G(V,E)
o Finite set V={n1, n2, …, nm} of nodes.o Finite set E={e1, e2, …, ep} of nodes.o Where each Ek={ni, nj} is an ordered pairo (start node, terminal node)
- For example: V={n1, n2, n3, n4},E= {e1, e2, e3}={(n1,n2), (n2,n4), (n1,n3)}
- Nodes of degreeso Indeg (ni) = number of distinct edges that have ni as their terminal node.o Outdeg (ni) = number of distinct edges that have ni as their start node.o Src node : a node with indeg = 0o Sink node: a node with outdeg = 0o Transfer node: a node with (indeg != 0 & outdeg != 0).
- Directed path:o a sequence of edges such that for any adjacent pairs of ei, ej, the terminal node of ei is
the start node of ej- Connectness: for two nodes ni, nj :
o 0-connected: iff there is no path between ni & njo 2-connected: iff there is a path between ni & njo 3-connected: iff there is a path between ni to nj or and a path between nj to ni.
- Strongly connected graph:o all pairs of nodes are 3-connected.
- Program graphs:o Given a pgm in an imperative language, its graph is a directed graph in which nodes are
entire statements or fragment of statements and edges represent flow of control.
n1
n2
n3
n4e1
e3
e2
)24(
A Brief Review of G raphs (Cont.)
Graphs for ctrl structures:
- Connected compounds are collapsed into individual nodes- A sequence of 2-connected nodes and a node with outdeg ≥ 2 can be mapped into a single
node:
- Areas bound by edges are called a region. You will see example later.- For compound conditions in a pgm, a separate node is created for each condition, so if (c1 and
c2) needs to be divided to two conditions if (c1) and if (c2). For example: (Look at handout 8).
C
T
TC CF FT C
If While Until (do/while)Sequence
Case
F
n2
n1
n3
)25(
A Brief Review of Graphs (Cont.)
McCabe's Cyclomatic Complexity Metrics:- Gives complexity- Give the number of linearly independent paths. (in each branch, pick up one of them, then go
over the other in the next round) number of test cases
- Formula:V(G) = #edges - #nodes +2
- You may count the designated node with/without its edges.- A complex example when there is a loop required repeating the cycle many times.
Example:
- Assuming N is the number of cycles that required the loop to be repeated:o When N=1, 5 pathso When N=2, 52 = 25 pathso When N=3, 53 = 125o When N=20, 51 + 52 + … + 520 = number of test cases need to be generated to cover all
the paths, which is impossible.
C
A
E
D
B
G
F
I
H
X return
20 times
)26(
Quiz on Graphs
EECS814: Software Quality AssuranceProfessor Hossein SaidianQuiz 1 – September 22, 2005 Name:
1. Draw a flow graph for the following code segment. For simplicity, assume that P1, P2, P3 are simple predicates:
S1;if (P1) {
for (i = 1; P2; i++) {if(P3)
S2;}If (P3) S3;
} elseS4;
S5;
)27(
2. Testing Techniques: Structural Testing
Introduction:- Complements functional testing.- Based on implementation- Powerful mathematical foundations
Representing graphs that are widely used for testing:- At the unit-level:
1. Program Graph (pgm graph):Given a program written in an imperative programming language, its program graph is a direct graph in which nodes are program statements, and edges represent flow of control (there is an edge from node i to node j iff the statement corresponding to node j can be executed immediately after the statement corresponding to node i)
- At the system-level behavior:2. FSM (Finite State Machine)
is a directed graph in which states are nodes and transitions are edges. Sources and sink states become initial and terminal nodes, paths are modeled as paths, and so on. All the real-time extensions of structured analysis use some form of finite state machine, and nearly all forms of object-oriented analyses require them.
3. Petri netsA Petri net is a bipartite directed graph (P, T, In, Out), in which P and T are disjoint sets of nodes, and In and Out are sets of edges, where , and
. Petri nets can do what FSM does, but not the other way round (such as parallelism).
Structural Testing Techniques- Control Flow-Based Testing:
1. Statement Coverage2. Branch Coverage3. Condition Coverage4. Branch / Condition Coverage5. Multi-Condition Coverage6. Independent Path Coverage
- Data Flow-Based Testing:1. All Definitions: each definition to a reachable use2. All Uses: definition to each reachable use3. All def-use: each definition to each reachable use
)28(
2.1 Structural Testing : Control Flow-Based Testing
1. Statement Coverage (aka line coverage):- Each executable statement is executed at least once- Advantage: reliable simple- Disadvantage: insensitive to certain ctrl structures- Example:
int *p = NULL;if (cond) p=&v;*p = 100;
- Without a test case to force a false evaluation, the code is considered covered.- Also, does not report whether loops reach termination condition; only whether the loop body is
executed.- Statement coverage is blind to other classes of errors:
1. Assignment blindness: due to assignment of a value to a variable, error does not propagate. For example:
if (cond) x=1;
if (x+y>=0) // should be (y>0)...
2. Equality blindness: due to an equality check of a variable, error does not propagate. For example, we choose a=0 to check the following statement:
if (a==0)...if (b>3) // should be (a+b>3)
...
3. Self-blindness: conditional covers ups the error. For example, we choose x=0 to check the following statement:
if(x<1) // should be (2*x<1)...
)29(
2. Branch Coverage:
- Each branch of a decision must be executed- Stronger than stmt coverage- The compound conditions consider one node unlike path coverage. (Look at the example in Branch /
Condition Coverage section).
- Path subsumes branch- Path subsumes statement
- Example: (Test cases to satisfy Branch converge)
if ((x>1) && (y==1))z=1;
elsez=0;
if ((x==2) || (y>1))z++;
elsez=-1;
)30(
Subsumption Definition:
Criteria C1 subsumes criteria C2, iff:- For all programs p being tested
with specifications s- All test sets t- t is adequate according to C1
for testing p with respect to s implies that t is adequate according to C2 for testing p
Test Case #X ValueY ValueStatements121z=1;z=-1;222z=0;z++;
Sep 29, 05
Example on Statement Converge and Branch Converge ( 'Count' Program) (Dependable Software Systems (Path Testing) @ Spiros Mancoridis)
The programming code for Count:
/* COUNTThis program counts the number of characters and lines in a text file.INPUT: Text FileOUTPUT: Number of characters and number of lines.
*/
1 main (int argc, char *argv [])2 {3 int numChars = 0;4 int numLines = 0;5 char chr;6 FILE *fp = NULL;78 if (argc < 2)9 {10 printf ("\nusage: %s <filename>", argv[O]);11 return (-1);12 }13 fp = fopen (argv[1], "r");14 if (fp == NULL)15 {16 perror (argv[l]); /* display error message */17 return (-2);18 }19 while (!feof(fp))20 {21 chr = getc (fp); /* read character */ .>22 if (chr == '\n' ) /* if carriage return */ y23 ++numLines;24 else25 ++numChars;26 }27 printf("\nNumber of characters = %d", numchars);28 printf("\nNumber of lines = %d" , numLines) ;29 }
Flowgraph for Count:
1 8 11 17 19 22 24 26 2914 23
)31(
Example on Statement Converge and Branch Converge (Cont.)
Test cases to satisfy Statement Coverage for Count:
1 8 11 17 19 22 24 26 29a
g
F
T
b
F
c
h
T
d e
f
j
i
k
TT
FF14 23
l
Test cases to satisfy Branch Coverage for Count:
PATHS PROCESS LINKS TEST CASES
a b c d e f g h i j k l INPUT OUTPUT
ab None “Usage: COUNT <filename<”
agc Invalid Input Filename
Error Message
aghdjkli
Input File with one character
and no Carriage Return at the
end of the line
Number of characters = 1
Number of lines = 0
aghdefli
Input file with no characters and
one carriage return
Number of characters = 0
Number of lines = 1
)32(
PATHS DECISIONS TEST CASES
8 14 19 22 INPUT OUTPUT
ab T None “Usage: COUNT <filename<”
agc F T Invalid Input Filename
Error Message
aghdjkli F F T,F F Input File with one character and no
Carriage Return at the end of the line
Number of characters = 1
Number of lines = 0
aghdefli F F T,F T Input file with no characters and one
carriage return
Number of characters = 0
Number of lines = 1
3. Condition Coverage:
- Each condition with a decision must assure all possible values.- Potentially stronger than branch coverage.- NOTE: you need at least to execute the true part.- Example:
if ((x>1) && (y==1))
Z=1;elseZ=0;
4. Branch / Condition Coverage:
- Try to cover all conditions and branches:- Example:
if (x<y && x>2)a++;
elsea=0;
if (x>0 && y>0)b++;
elseb=0;
x<y
Tx>2
a++a=0
T
y>0
Tx>0
b++b=0
T Paths not covered in branch/condition coverage
)33(
Test Case #X ValueY ValueStatements121z=1;201z=0;
Test Case #X ValueY ValueStatements134a++;b++;2-3-4a=0;b=0;
Example on Statement Converge , Branch Converge , and Condition Converge
Programming Code:
if (a > 3)x++;
if (b==3)y=0;
Flowgraph:
Test Cases:
Coverage Type Paths Test CasesStatement Coverage 4 (a=6, b=3)
Branch Coverage 14
(a=1, b=1)(a=6, b=3)
Condition Coverage 4 (a=6, b=3)
Branch/Condition Coverage 14
(a=1, b=1)(a=6, b=3)
a>3
x++
y=0
b=3
T
T
Path #4 Path #3
)34(
Path #1Path #2
5. Multi-Condition Coverage:
- subsume condition & branch coverage- Every combination of possible conditional values within a decision is chosen, so if we have if(C1 & C2
& C3 & C4), we will have 16 possibilities.- usually better than branch cover- Differences between condition, branch, condition/branch, multi-condition coverage:
if (a && b)X();
else Y();
1. Condition Coverage: if (True && False) → Y(); if (False && True ) → Y();
2. Branch Coverage: if (True && True ) → X(); if (False && True ) → Y();
3. Branch / Condition Coverage: if (True && True ) → X(); if (False && False) → Y();
4. Multi-conditions Coverage: if (True && True ) → X(); if (True && False) → Y(); if (False && True ) → Y(); if (False && False) → Y();
6. Basic Path Testing:
- The cyclomatic complexity of a connected directed graph represents the # of linearly independent cycles.
- To determine a set of basic paths:1. Draw a flow graph2. Pick a baseline (corresponding to normal execution)3. Succeeding paths: retrace the baseline until a decision point is reached; flip the decision
and continue as much as the baseline as possible4. Repeat until all decision have been flipped
1ConditionCoverage
2Branch
Coverage3
4
)35(
Example on Condition Converge and Multi-Condition Converge
Programming Code:
if (a>0 && c==1)x++;
if (b==3 || d<0)y=0;
Flowgraph:
Test Cases:
1. Condition Coverage: Test the True and False condition of each expression.Test
Case #Values Statementsa b c d
1 True 1 True 3 True 1 True -1 x++; y=0;2. Multi-Condition Coverage: Covers all the possibilities.
TestCase #
Valesa b c d Statements
1 True 1 True 3 True 1 True -1 x++; y=0;2 False 0 True 3 True 1 False 13 True 1 False 4 False 2 True -14 False 0 False 4 False 2 False 1
a>0
Tc=1
x++
T
b=3
d<0
T
y=0
T
Paths not covered in branch/condition coverage
)36(
Example on Path Converge ('Average' Program)
The programming code for Average:
/* AVERAGEThis procedure computes the average of 100 or fewer numbers that lie between bounding values; it also computes the sum and the total number valid.*/
PROCEDURE average;
INTERFACE RETURNS average, total.input, total.valid;INTERFACE ACCEPTS value, minimum, maximum;TYPE value[l:100] IS SCALAR ARRAY;TYPE average, total.input, total.valid, minimum, maximum, sum IS SCALAR;TYPE i IS INTEGER;
i = 1;total.input = total.valid = 0;sum = 0;
DO WHILE value[i]<>-999 AND total.input<100
increment total.input by 1
IF value[i]>=minimum AND value[i]<=maximum
THEN increment total.valid by 1;sum = s sum + value[i]
ELSE skipENDIFincrement i by 1:
ENDDO
IF total.valid>0THEN average=sum / total.valid;ELSE average =-999;
ENDIFEND average
Test cases to satisfy Path Coverage for Average:
# of independent path = # of complexity = # of regions = #e-#n+2 = of predicates +1 = 6
# Paths Test Cases1 1, 2, 10, 11, 13 (i=-999, total.valid=1)2 1, 2, 10, 12, 13 (i=-999, total.valid=0)3 1, 2, 3, 10, … (i=1, total.input=1000)4 1, 2, 3, 4, 5, 6, 7, 8, 9, 2 … (i=1, total.input=10, maximum≥i≥minimum)5 1, 2, 3, 4, 5, 6, 8, 9, 2, … (i=1, total.input=10, maximum<i≥minimum)6 1, 2, 3, 4, 5, 8, 9, 2, (i=1, total.input=10, maximum<i<minimum)
)37(
6 RegionsPredicate
T
1
F2
3
4
5
6
8 7
9
10
12 11
13
F
F
F T
I
II
III
IV
V
VI
1
2 3
45 6
7
8
9
10
11112
13
Example on Path Converge ('TotalPrice' Program)
The programming code for TotalPrice:
1 rushcharge = 01 if nextday = 'Y'2 rushcharge = 14.503 endif3 tax = amount * 0.083 if amount >= 10004 shipcharge = amount * .06 + rushcharge5 else if amount >= 2006 shipcharge = amount * .08 + rushcharge7 else if amount >= 1008 shipcharge = 10 + rushcharge9 else if amount >= 5010 shipcharge = 9 + rushcharge11 else if amount >= 2512 shipcharge = 7 + rushcharge13 else13 shipcharge = 5 + rushcharge14 endif14 total = amount + tax + shipcharge
Test cases to satisfy Path Coverage for TotalPrice:
# of independent path = 7
# Path NextDay AMT1 1, 2, 3, 5, 7, 9, 11, 13,
14'Y' AMT < 25
2 1, 3, 4, 14 'N' AMT ≥ 1000 (e.g. 1500)3 1,3,5,6,14 'N' 200 ≤ AMT < 1000 (e.g. 500)4 1,3,5,7,8,14 'N' 100 ≤ AMT < 200 (e.g. 150)5 1,3,5,7,9,10,14 'N' 50 ≤ AMT < 100 (e.g. 75)6 1,3,5,7,9,11,12,14 'N' 25 ≤ AMT < 500 (e.g. 35)7 1,3,5,7,9,11,13,14 'N' AMT < 25 (e.g. 15)
)38(
1
2
3
T
45
T
67
T
89
T
1011
T
1213
T
14
Conclusion Example about Control Flow-Based Coverage
The programming code:
123456789101112131415
function P return INTEGER isbegin
X, Y: INTEGER;READ (X); READ (Y);while (X > 10) loop
X := X - 10;exit when X = 10;
end loop;if (Y < 20 and then X mod 2 = 0) then
Y := Y + 20;else
Y := Y - 20;end if;return 2 X + Y;
end P;
Flowgraph:
Test Cases:
Coverage Types # Paths Test CasesStatement Coverage 1 {4,5,6,7,9,9',10,14} (x=20, y=10)Branch Coverage(NOTE: 9 and 9` is consider one node)
12
{4,5,6,7,9,9',10,14}{4,5,6,7,5,9,12,14} (x=20, y=10)
(x=15, y=30)
Condition Coverage(NOTE: you need at least to execute the TRUE part)
123
{4,5,6,7,9,9',10,14}{4,5,9,12,14}{4,5,6,7,5,9,9`,12,14}
(x=20, y=10)(x=5, y=30)(x=21, y=10)
Path Coverage
12345
{4,5,9,12,14}{4,5,6,7,5,9,12,14}{4,5,6,7,5,9,9`,12,14}{4,5,6,7,9,9`,10,14}{4,5,6,7,8,12,14}
(x=5, y=30)(x=15, y=15)(x=21, y=10)(x=20, y=10)(x=15, y=30)
10
9` 14
12
F
T95
76
2,3,4 FT
)39(
2.2 Structural Testing : Data Flow-Based Testing 2.2.1 Define/Use Testing
Introduction:- Many failures involve because of execution of an incorrect variable definition:
o Incorrect assignmento Predicate is faultyo Definition is missing
- Explore sequence event related to the data (variable) state
- Examples:
Read (V1, V2, … Vn)o Definition V1, V2, … Vn
Write (V1, V2, … Vn)o C-USE of V1, V2, … Vn
Method(f1, f2, … fn)o Definition of each formal parameter.
while B do So Predicate use of variables in B
for (V=e1 to e2)o Compute use of each variable in e1 & e2o Definition of Vo Predicate use of V
if B then S1; else S2o Predicate use (P-USE) in B
case e1o Predicate use (P-USE) in e1
Dataflow Anomalies:- Suspicious and data in a program
o Not errors to be detected by the compilero Represent potential problem or bad programming style
- Example of anomalies:o live variable problem; a variable is defined twice before its used:
X = a; : :X = b;
o A variable is defined but never usedo A variable is used but never definedo Unreferenced labelso Unreachable code
)40(
Annotations:
- d: defined, created, initialized (data decln, assignment, input)
- k: Killed, undefined, released- u: used for something
o computationo predicates
- v: expressiono C-USE of all
variables in
no use of X
Oct 20, 05
Normal Dataflow Condition:
- du- kd- uu- _d (last definition)- k_ (ok, last thing done)- u_ (ok, last use)
Example on Normal Dataflow Condition:
A
K
U D
k u, kd
d
ud, ku k, u, d
)41(
Dataflow Anomalies
dd define followed by define *
dk defined then killedku killed then used__ nothing happens_k nothing to kill_u undefinedd_ defined but not usedu_ normal unless the
variable should have been killed
* if a definition comes after
Define X
Use YKill Z
Define YUse Z
Use YUse Z
Kill YDefine Z
Kill ZUse X
Define Z
Define XUse XUse Z
Path 1 Path 2
Respect to X- Path 1 (the concern about define X in 2nd node)
o _d (normal)o dd (concern)o du (normal)
- Path 2o du (normal)
Respect to Y- Path1
o _u (bad)o ud (ok)o du (normal)o uk (normal)
- Path 2o _u (bad)o ud (ok)o dk (probable error)
Respect to Z- Path1
o _k (problem)o ku (problem)o uu (normal)o ud (normal)
- Path 2o _k (problem)o kk (probable problem)o kd (normal)o du (normal)o uu (normal)o ud (normal)
Dataflow Definition:
- G(P) program grapho P programo V P's Variables
- PATH (P)o Paths in P
- DEF (v,n) <variable, node>o v Vo n G(P)o v defined at n
- USE (v,n)o v Vo n G(P)o v is used at n
- du-path (definition-use)o A path in PATH(P) such that for some v V, there is DEF(v,m) & USE(v,n) such that m & n
are initial and final nodes of the path.- dc-path (definition-clean path)
o For a v V, dc-path is a path PATH (P) with initial & final nodes DEF(v,m), USE(v,n), such that no other node in the path is a defining node of V.
Example 1:
1 i :=0;2 while (i<2)3 {4 cout << "in loop";5 i++;6 }
Example 2:
1 sum :=0;2 read(n); 3 i=1; 4 while (i <= n) { 5 read (num); 6 sum += num 7 i++; 8 }9 print(sum)
)42(
Types of USE(v,n)
P-use: # of out-degree ≥ 2C-use: # of out-degree ≤ 1
Variable DEF USE DU-Paths
i
1 2 <1-2>1 5 <1-2-3-4-5>5 2 <5-6-2>5 5 Disallowed
Variable DEF USE DU-Paths TestCases *
n 2 4 <2-3-4> n=0num 5 6 n=5
sum
1 6 n=51 9 n=06 6 Disallowed n=56 9 n=5
i
3 4 n=03 7 n=57 4 n=57 7 Disallowed n=5
* Note: Two sets of data cover all the DU-Paths: n=0 and n=5
Example on Define/Use Testing ('Commission' Program)
The programming code for Commission:
1 Program Commission (INPUT,OUTPUT)
2 Dim locks, stocks, barrels As Integer3 Dim lockPrice, stockPrice, barrelPrice As Real4 Dim totalLocks, totalStocks, totalBarrels As Integer5 Dim IockSales, stockSales, barrelSales As Real6 Dim sales, commission As Real
7 lockPrice = 45.08 stockPrice = 30.09 barrelPrice = 25.010 totalLocks = 011 totalStocks = 012 totalBarrels = 0
13 Input(locks)14 While NOT(1ocks = -1) 'loop condition uses -1 to indicate end of data15 Input(stocks, barrels)16 totalLocks = totalLocks + locks17 totalStocks = totalStocks + stocks18 totalBarrels = totalBarrels + barrels19 Input(stocks)20 EndWhile
21 Output("Locks sold: ", totalLocks)22 Output("Stocks sold: ", totalStocks)23 Output("Barre1s sold: ", tota1Barrels)24 lockSales = lockPrice * totalLocks25 stockSales = stockPrice * totalStocks26 barrelSales = barrelPrice * totalBarrels27 sales = lockSales+stockSales+barrelSales28 Output("Total sales: ", sales)
29 If (sales > 1800.0)30 Then31 commission = 0.10 * 1000.032 commission = commission + 0.15 * 800.033 commission = commission + 0.20*(sales-
1800.0)34 Else If (sales > 1000.0)35 Then36 commission = 0.10 * 1000.037 commission = commission + 0.15*(sales-
1000.0)38 Else commission = 0.10 * sales39 EndIf40 EndIf
41 Output("Commission is $", commission)42 End Commission
)43(
7-13
14
15-19
20-28
29
30-33 34
39
35-37 38
40
41-42
Highlight DEF/USE Variables
DEF: Italic & UnderlineP-use: Underline
Example on Define/Use Testing ('Commission' Program) (Cont.)
DEF/USE Variables in Commission:Variable DEF of node USE at nodelockPrice 7 24stockPrice 8 25barrelPrice 9 26totalLocks 10,16 16,21,24totalStocks 11,17 17,22,25totalBarrels 12,18 18,23,26Locks 13,19 14,16Stocks 15 17Barrels 15 18lockSales 24 27stockSales 25 27barrelSales 26 27Sales 27 28,29,33,34,37,38commission 31,32,33,36,37,38 32,33,37,41
DEF/USE Paths:# Variable DEF
NodeUSE
NodeDEF
ClearDu-Paths
1 lockPrice 7 24 Yes P1=<7,8,9,10,11,…,21,22,23,24>2 stockPrice 8 25 Yes P2=<8,9,10,11,12,…,22,23,24,25>3 barrelPrice 9 26 Yes P3=<9,10,11,12,13,…,23,24,25,26>4 totalLocks 10 16 Yes P4=<10,11,12,13,14,15,16>5 totalLocks 10 21 No P5=<P4,17,18,19,20,21>6 totalLocks 10 24 No P6=<P5,22,23,24>
Note: Ignoring the possible repetition of the while-loop
7 totalLocks 16 16 Yes P7=<16,16> or<16,17,18,19,20,14,15,16>Disallowed as du-paths
8 totalLocks 16 21 Yes P8=<16,17,18,19,20,21>9 totalLocks 16 24 P9=<P8,22,23,24>10 totalStocks 11 17 Yes P10=<11,12,13,14,15,16,17>11 totalStocks 11 22 No P11=<P10,18,19,20,21,22>12 totalStocks 11 25 No P12=<P11,23,24,25>13 totalStocks 17 17 Yes Disallowed as du-paths14 totalStocks 17 22 No P14=<17,18,19,20,21,22>15 totalStocks 17 25 No P15=<P14,23,24,25>16 totalBarrels 12 18 Yes P16=<12,13,14,15,16,17,18>17 totalBarrels 12 23 No P17=<P16,19,20,21,22,23>18 totalBarrels 12 26 No P18=<P17,24,25,26>19 totalBarrels 18 18 Yes Disallowed as du-paths20 totalBarrels 18 23 Yes P20=<18,19,20,21,22,23>
)44(Oct 27, 05
Example on Define/Use Testing ('Commission' Program) (Cont.)
21 totalBarrels 18 26 Yes P21=<P20,24,25,26>22 Locks 13 14 Yes P22=<13,14>23 Locks 13 16 Yes P23=<P22,15,16>24 Locks 19 14 Yes P24=<19,20,14>25 Locks 19 16 Yes P25=<P24,15,16>26 Stocks 15 17 Yes P26=<15,16,17>27 Barrels 15 18 Yes P27=<P26,18>28 lockSales 24 27 Yes P28=<24,25,26,27>29 stockSales 25 27 Yes P29=<25,26,27>30 barrelSales 26 27 Yes P30=<26,27>31 Sales 27 28 Yes P31=<27,28>32 Sales 27 29 Yes P32=<P31,29>33 Sales 27 33 Yes P33=<P32,30,31,32,33>34 Sales 27 34 Yes P34=<P32,34>35 Sales 27 37 Yes P35=<P34,35,36,37>36 Sales 27 38 Yes P36=<P32,38>37 commission * 31 32 Yes P37=<31,32>38 commission 31 33 Yes P38=<31,32,33>39 commission 31 37 n/a Not feasible40 commission 31 41 Yes P40=<31,32,33,40,41>41 commission 32 32 Yes Disallowed as du-paths42 commission 32 33 Yes P42=<32,33>43 commission 32 37 n/a Not feasible44 commission 32 41 Yes P44=<32,33,40,41>45 commission 33 32 n/a Not feasible46 commission 33 33 Yes Disallowed as du-paths47 commission 33 37 n/a Not feasible48 commission 33 41 Yes P48=<33,40,41>49 commission 36 32 n/a Not feasible50 commission 36 33 n/a Not feasible51 commission 36 37 Yes P51=<36,37>52 commission 36 41 Yes P52=<36,37,39,40,41>53 commission 37 32 n/a Not feasible54 commission 37 33 n/a Not feasible55 commission 37 37 Yes Disallowed as du-paths56 commission 37 41 Yes P56=<37,39,40,41>57 commission 38 32 n/a Not feasible58 commission 38 33 n/a Not feasible59 commission 38 37 n/a Not feasible60 commission 38 41 Yes P60=<38,39,40,41>
* Note: In commission variable, you may consider only the real ones
)45(
DU-Path Test Coverage Metrics (Rapps-Weyuker dataflow metrics)
- All-Defs Criterion: For every variable v V, T contains def-clear paths from every defining node of V to a use of V
- All-Uses Criterion: For every variable v V, T contains def-clear paths from every defining node of v to every use of v, and to successor node of each USE(v,n)
- All-P-Uses/Some-C-Uses: For every variable v V, T contains def-clear paths from every defining node of v to every predicate use of v; and if a definition of v has no P-uses, a def-clear leads to at least one C-use
- All-C-Uses/Some-P-Uses:For every variable v V, T contains def-clear paths from every defining node of v to every C-use of v; and if a definition of v has no C-uses, a def-clear path leads to at least one P-use
- All-DU-Paths: For every variable v V, T contains def-clear paths from every defining node of v to every use of v and to successor node of each USE(v,n) and that these paths are either single loop traversals or they are cycle.
Subsumption Graph
All-Paths
All-DU-Paths
All-Uses
All-C-Uses/Some-P-Uses All-P-Uses/Some-C-Uses
All-Defs All-P-Uses
All-Edges
All-Nodes
)46(
Annotations:
P ProgramG(P) Graph of PV Variables in
P
Frankl and Weyuker's seven data flow criteria
- All-defs:requires that for each definition of a variable X in P, the set of paths P executed by the test set T contains a definition-clear subpath from the definition to at least one c-use or one p-use of X.
- All-c-uses:requires that for each definition of X in P, and each c-use of X reachable from the definition, Pi contains a definition-clear subpath from the definition to the c-use.
- All-p-uses: requires that for each definition of X in P, and each p-use of X reachable from the definition, Pi contains a definition-clear subpath from the definition to the p-uses.
- All-c-uses/some-p-uses:requires that for each definition of X in P, if there exists at least one c-use of X reachable from the definition, Pi contains a definition-clear subpath from the definition to at all reachable c-uses of X, otherwise, Pi contains a definition-clear subpath from the definition to a reachable p-use of X.
- All-p-uses/some-c-uses:requires that for each definition of X in P, if there exists at least one p-use of X reachable from the definition, Pi contains a definition-clear subpath from the definition to all reachable p-uses of X, otherwise, Pi contains a definition-clear subpath from the definition to a reachable c-use of X.
- All-uses: requires that for each definition of X in P, Pi contains a definition-clear subpath from the definition to all reachable c-uses and p-uses of X.
- All-du-paths: requires that for each definition of X in P, Pi contains all definition-clear subpaths from the definition to all reachable c-uses and p-uses of X, such that each subpath contains no loops, or contains one complete loop.
In addition: All-paths require that all paths through the program be executed.
)47(
2.2 Structural Testing : Data Flow-Based Testing 2.2.2 Slice-Based Testing
Introduction:- A program slice is a subset of a pgm- Enables programmers to view subsets of a pgm by filtering out code that is not relevant.- Why is it useful?
o More manageable.o When testing, debugging, or understanding, most code is irrelevant.o Pgm slicing provides a convenient way to filtering out irrelevant codeo Slices can be computed by static analysis.
- Definitions:o P is a pgmo V is a subset of variables at line #no A slice S(v,n) produces a portion of a pgm that contributes to the value of v just before
statement at n.o S(v,n) must be derived from P (by deleting statements)o S(v,n) must be syntactically correct.o For all execution of P, the value of v in the execution of S(v,n) just before location n must
be the same value of in execution of P just before n.
Program Slicing Process:1. Select the slicing criteria:
a. a variable (or set of variables)b. a pgm location
2. Generate slices (may use such as tools: spyder, unravel, …)3. Perform testing on slices4. Merge modified with the rest of the modified slices back into the original pgm
)48(Nov 3, 05
Example on Slice-Based Testing
The programming code:
0 main() {1 int mx, mn, av;2 int tmp, sum, num;34 tmp = readInt() :5 mx = tmp;6 mn = tmp;7 sum = tmp;8 num = 1;910 while(tmp >= 0)11 {12 if (mx < tmp)13 mx = tmp;14 if (mn > tmp)15 mn = tmp;16 sum += tmp;17 ++num;18 tmp = readInt() ;19 }2021 av = sum / num;22 printf("\nMax=%d", mx) ;23 printf("\nMin=%d", mn);24 printf("\nAvg=%d", av);25 printf("\nSum=%d", sum);26 printf("\nNum=%d", num) ;27 }
)49(
Slice S (num, 26):0 main() {2 int tmp, num;4 tmp = readInt( ):8 num = 1;10 while(tmp >= 0)11 {17 ++nun;18 tmp = readInt( );19 }26 printf (" \nNum=%d", num);27 }
Slice S (sum, 25):0 main() {2 int tmp, num;4 tmp = readInt( ):7 sum = tmp;10 while(tmp >= 0)11 {16 sum += tmp;18 tmp = readInt( );19 }25 printf("\nSum=%d", sum);27 }
Slice S (av, 24):0 main() {1 int mx, mn, av;2 int tmp, num;4 tmp = readInt( ):7 sum = tmp;8 num = 1;10 while(tmp >= 0)11 {16 sum += tmp;17 ++num;18 tmp = readInt( );19 }21 av = sum / num;24 printf("\nAvg=%d", av);
Slice S (mn, 23):0 main() {1 int mx, mn, av;2 int tmp, num;4 tmp = readInt( ):6 mn = tmp;10 while(tmp >= 0)11 {14 if (mn > tmp)15 mn = tmp;18 tmp = readInt( );19 }23 printf("\nMin=%d", mn);27 }
Slice S (mx, 22):0 main() {1 int mx, mn, av;2 int tmp, num;4 tmp = readInt( ):5 mx = tmp;10 while(tmp >= 0)11 {12 if (mx < tmp)13 mx = tmp;18 tmp = readInt( );19 }22 printf("\nMax=%d", mx);27 }
Mutation Testing
Mutation Testing:- A testing techniques that focuses on measuring the adequacy of test cases- Not a testing strategy- Should be used in conjunction with traditional techniques- Goal: cause the mutant programs to fail (thus showing the effectiveness of test cases)- Faults are introduced into the pgm by:
1. Creating many versions of pgm called mutants2. Each mutant contains a single fault.3. Test cases are applied to the original pgm & mutants4. If the original pgm of all mutants generate the same output, the test case is inadequate.
- A test case is shown to be adequate by finding at least one mutant that generates a different output.
- Theoretical analysis as well experiments result show that it is an effective approach to measure the adequacy of test cases
Categories of Mutation Operators:Category Examples
Original PGM MutationOperand Replacement if( x > y) (x>5)
(6>y) (y>x)
Operator Replacement a=b*(c-d) b+(c-d) b/(c-d)
Expression ModificationSuch as: Replace an operator or insert a new one
if (x==y) (x>=y) (x==++y)
Statement ModificationSuch as: Delete else part / delete if-then
if (x==y) a=5;else a=1;
if(x==5)a=5;
Mutation Testing Process:1. Generate test cases2. Run each test case against the original pgm
a. If output is incorrect, the pgm must be modified & re-testedb. If output is correct go to next step
3. Construct mutants (using a tool like Mothra)4. Execute each test case against a alive mutant
a. If output of mutant differs from the original pgm, kill the mutant.
Mutation Score:- A mutation score of a set of test cases is the percentage of non-equivalent mutants killed.- (we need it to be 100%)
where D is dead mutantsN is # of mutantsE is # of equivalent mutants
)50(
2.3 Structural Testing : Retrospective on Structure Testing
Observation:- DF: for pgms that are computationally intensive- Path/cyclomatic metric are more criteria metrics, not procedures:
o e.g., Basis Path gives a lower boundary on how much testing is necessary.o Slice composition should be used to redevelop difficult sections of codeo Statement & Branch Coverage must be an absolute minimum (mandatory?) testing reg.
When to stop testing?1. When you run out of time.2. When continued testing causes no new failures.3. When continued testing reveals no new faults.4. When you cannot think of any new test cases.5. When you reach a point of diminishing returns.6. When mandated coverage has been attained.7. When all faults have been removed.
Trend of test coverage items:
Trend of test method effort:
)51(
Types of Testing DON'T Like to See
During a particularly long and painful bout of REGRESSION TESTING our application software, my buddies and I came up with this list of other types of testing we'd like not to see:
AGGRESSION TESTING: If this doesn't work, I'm gonna kill somebody.
CONFESSION TESTING: Okay, Okay, I did program that bug.
CONGRSSIONAL TESTING: Are you now, or have you ever been a bug?
DEPRESSION TESTING: If this doesn't work, I'm gonna kill myself.
EGRESSION TESTING: Uh-oh, a bug... I'm outta here.
DIGRESSION TESTING: Well, it works, but can I tell you about my truck...
EXPRESSION TESTING: #@%^&*!!!, a bug.
OBSESSION TESTING: I'll find this bug if it's the last thing I do.
OPRESSION TESTING: Test this now!
POISSION TESTING: Alors! Regardez le poission!
REPRESSION TESTING: It's not a bug, it's a feature.
SECCESSION TESTING: The bug is dead! Long lives the bug!
SUGGESTION TESTING: Well, it works but wouldn't it be better if...
COMPRESSION TESTING: []
)52(
3. Integration Testing
Introduction:- Testing activities that integrate software components together.- Definition: is a systematic technique for constructing the software architecture while at the same
time conducting tests to uncover errors associated with interfacing.- Objective: to take unit tested components and build a program structure that has been dictated
by design.- Several approaches:
1. Decomposition-Based Integration: using the functional decomposition tree.a. Top-down Integrationb. Bottom-up Integrationc. Sandwich Integration: a combination of top-down and bottom-up integration.
2. Call Graph-Based Integration: using a direct grapha. Pair-Wise Integrationb. Neighborhood Integration
3. Path-Based Integration (MM-Path Based Integration)
Driver vs. Stub:- Driver: is the main program that accepts test case data, passes such data to the component (to
be tested), and prints relevant results.- Stubs: serve to replace modules that are subordinate to (called by) the component to be tested.- Drivers and stubs represent overhead because both are software that must be written without
delivering with the final software product.
Figure: Driver vs. Stub (pressman, 2005)
)53(
3.1 Integration Testing: Decomposition-Based Integration
1. Top-down Integration:- What are the steps for top-down integration?
1. The main control module is used as a test driver and stubs are substituted for all components directly sub-ordinate to the main control module.
2. Depending on the integration approach selected ( i.e., depth or breadth first), sub-ordinate stubs are replaced one at a time with actual components.
3. Tests are conducted as each component is integrated.4. On completion of each set of tests, another stub is replaced
with the real components.5. Regression testing may be conducted to ensure that new
errors have not been introduced.The process continues from step 2 until the entire
program structure is built.
- What problems may be encountered when top-down integration is chosen?1. Delay many tests until stubs are replaced with actual modules.2. Develop stubs that perform limited functions that simulate the actual module, or3. Integrate the software from the bottom of the hierarchy upward.
- Number of stubs required:Stubs = Number of Nodes – 1 (the driver)
2. Bottom-up Integration:- What are the steps for bottom-up integration?
1. Low-level components are combined into clusters (sometimes called builds) that perform a specific software sub-function.
2. A driver (a control program for testing) is written to coordinate test case input and output.
3. The cluster is tested. 4. Drivers are removed and clusters are combined moving
upward in the program structure.
- Number of drivers required:Drivers = Nodes - Leaves
)54(
3.2 Integration Testing: Call Graph-Based Integration
Introduction- Move away from a purely structural basis toward a behavioral basis.- Eliminate the stub/driver development effort.
1. Pair-Wise Integration:- Use actual code instead of stubs/drivers.- Cover each edge.- Restrict a test session to two pair of units (not big bang).
2. Neighborhood Integration:- A neighbor is any node connected.
o A set of nodes that are one edge away from the given node
o All intermediate predecessors (drives) and all immediate successors (stubs).
- Pro: reduce the # of tests- Con: problem isolating especially in large neighborhoods.
- Computing the number of neighborhoods:
Neighborhoods = interior nodes + source nodes
where Interior nodes = nodes – (source nodes + sink nodes)
Neighborhoods = nodes - sink nodes
)55(
3.3 Integration Testing: Path-Based Integration
Introduction:- Combined unit testing and integration testing.- When a unit executes, some path of source statements is traversed.- Suppose that a call goes to another unit along such a path:
o Unit testing: suppress the call statement because control eventually returns to the calling unit anyway.
o Integration testing: abandon the single-entry, single-exit percept and treat such calls as an exit followed by an entry.
- We need to refine some of the program graph concepts.
New Definition:- Source node:
o A statement fragment at which some program execution begins or resumes.o Each module will have at least one.o The first executable statement in a unit is a source node
- Sink node:o A statement fragment at which program execution ends.o E.g. final statement in a program.
- Module Execution Path (MEP):o A sequence of statements that begins with a source node
and ends with a sink node with no intervening sink node.o Program graph may have multiple source and sink nodes
- Message:o A programming language mechanism by which one unit transfers control to another unit.
- MM-Path:o Module to module path (recall DD-Path: decision to decision path)o Interleaved sequence of Module Execution Paths (MEP) and messageso An example of Module Execution Paths (MEP) and MM-Path:
MEP(A,1) = <1, 2, 3, 6> MEP(A,2) = <1, 2, 4>MEP(A,3) = <5, 6>MEP(B,1) = <1, 2>MEP(B,2) = <3, 4>MEP(C,1) = <1, 2, 4, 5>MEP(C,2) = <1, 3, 4, 5>
- MM-Path Graph:o Given a sequence of units, MM-Path graph is a directed
graph in which nodes are modules, execution paths and edges correspond to messages and returns from one unit to another.
o Assumption, in integration testing, unit testing has already been done.
o The front figure shows an example of MM-Path Graph.
)56(Nov 10, 05
Hints in MM-Path:
Nodes are ModulesEdges are
An MM-Path
Path-Based Integration Pros:
1. Hybrid of functional and structural testing.
2. Coupled with actual system behavior.
3. Cost by elimination of stub
Example on Integration Testing
The programming code:
1 Main integrationNextDateType Date
Month As IntegerDay As IntegerYear As Integer
EndTypeDim today As DateDim tomorrow As Date
2 GetDate(today) 'msg13 PrintDate(today) 'msg24 tomorrow = IncrementDate(today) 'msg35 PrintDate(tomorrow) 'msg46 End Main
7 Function isLeap(year) Boolean8 If bear divisible by 4)9 Then10 If (year is NOT divisible by 100)11 Then isLeap = True12 Else13 If bear is divisible by 400)14 Then isLeap = True15 Else isLeap = False16 EndIf17 EndIf18 Else isLeap = False19 EndIf20 End (Function isleap)
21 Function lastDayOfMonth(month, year) Integer22 Case month Of23 Case 1: 1,3,5,7, 8, 10, 1224 lastDayOfMonth = 3 125 Case 2: 4,6,9, 1126 lastDayOfMonth = 3027 Case 3: 228 If (isLeap(year)) 'msg529 Then IastDayOfMonth = 2930 Else 1astDayOfMonth = 2831 EndIf32 EndCase33 End (Function IastDayOfMonth)
)57(
Example on Integration Testing (Cont.)
34 Function GetDate(aDate) Datedim aDate As Date
35 Function ValidDate(aDate) Boolean 'within scope of GetDate36 If ((aDate.Month > 0) AND (aDate.Month <=12)37 Then monthOK = True38 Else monthOK = False39 EndIf40 If (monthOK)41 Then42 If ((aDate.Day > 0) AND
(aDate.Day <= lastDayOfMonth(aDate.Month, aDate.Year))'msg643 Then &yOK = True44 Else dayOK = False45 EndIf46 EndIf47 If ((aDate.Year > 1 8 11) AND (aDate.Year <= 20 12)48 Then yearOK = True49 Else yearOK = False50 EndIf51 If (monthOK AND dayOK AND yearOK)52 Then ValidDate = True53 Else ValidDate = False54 EndIf55 End (Function ValidDate)
' GetDate body begins here56 Do57 Output("enter a month")58 Input(aDate.Month)59 Output("enter a day")60 Input(aDate.Day)61 Output("enter a year")62 Input(aDate.Year)63 GetDateMonth = aDate.Month64 GetDate-Day = aDate.Day65 GetDate.Year = aDate.Year66 Until (ValidDate(aDate)) 'msg767 End (Function GetDate)
68 Function IncrementDate(aDate) Date69 If (aDate.Day < lastDayOfMonth(aDate.Month)) 'msg870 Then aDate.Day = aDate.Day + 171 Else aDate.Day = 172 If (aDate.Month = 12)73 Then aDate.Month = 174 aDate.Year = aDate.Year + 175 Else aDate.Month = aDate.Month + 176 EndIf77 EndIf78 End (IncrementDate)
)58(
Example on Integration Testing (Cont.)
1. Functional decomposition of integration version:
2. Call graph of integration version:
3. MM-Path Based Integration (for May 27, 2002):
Main (1, 2)msg1GetDate (34, 56, 57,58, 59,60,61, 62, 63, 64, 65, 66)
msg7ValidDate (35, 36,37, 39,40, 41, 42))
msg6lastDayOfMonth (21,22,23,24,32, 33)'point of message quiessence
ValidDate (43.45,46,47,48, 50, 51, 52, 54, 55)GetDate (67)
'Main (3)
)59(
They will cause problem with pairwise integration because they are never directly called by Main program.
Static Program Testing Analysis: 1. Hoare Logic
Introduction:- Hoare logic (also known as Floyd–Hoare logic) is a formal system developed by the British
computer scientist C. A. R. Hoare, and subsequently refined by Hoare and other researchers. It was published in Hoare's 1969 paper "An axiomatic basis for computer programming". The purpose of the system is to provide a set of logical rules in order to reason about the correctness of computer programs with the rigour of mathematical logic.
- A programming language specification consists of a syntactic description and a semantic description.
o Syntactic description: symbols we can use in a languageo Semantic Description: what phrases in a programming language mean.
- In specific phrase, using natural language to describe three things: 1. Pre-conditions2. Invariant3. Post-conditions
Prototype:{P} C {Q}
where P and Q are assertions and C is a command. P is called the precondition and Q the postcondition. Assertions are formulas in predicate logic.
Two Types:- Partial Correctness: if C does not terminate, then there is
no "after", so Q can be any statement at all. Indeed, one can choose Q to be false to express that C does not terminate.
- Total Correctness: if C terminates and at termination Q is true, the expression exhibits. Termination would have to be proved separately.
Loop Invariant:- A constant (unchanging) predicate (constraint, fact)- Unaffected by group of mathematical operations (program statements) under consideration
o often the important identification of the program (states? the acceptance? look a little deeply)
- Establishing a loop invariant:o Define a predicate I that shows the logical relationship between i, s, and b
I: 1 ≤ i ≤ 11 ^ s = sum(b[k]) k=0, ..., i-1\\ i is changing
o Show that I is true before the loop and after each iteration of the loop so that it is true after the completion of the loop
o If I is true at these place, together with falsity for loop guard, show that the post condition holds
)60(
{P} C {Q} reads:
Whenever P holds of the state before the execution of C, then Q will hold
Example on Hoare Logic
Objective:A program that stores in the sum of array b[0,…,10] in S
The predicate I:I: 1 ≤ i ≤ 11 ^ s = sum(b[k]) k=0, ..., i-1
The programming code with comments:
i:=1;s:= b[0];
while (i<11)s:=s+b[i];i:=i+1;
end while
Precondition: True
(1) I is true
(2) is true
(3)I is truepostcondition: s = sum(b[k]) k=0,…, 10
Prove that I is True all the times:1. Before the loop:
i=1, s=b[0]
Does this affect I?I : 1 ≤ i ≤ 11 ^ s = sum(b[k]) k=0, ..., i-1 1 ≤ 1 ≤ 11 ^ b[0]= sum(b[k]) k=0,i-1 True ^ True True
2. Inside the loops = s + b[i];i = i + 1;
Do these affect I?I : 1 ≤ i ≤ 11 ^ s = sum(b[k]) k=0, ..., i-1
1 ≤ 1 < 11 ^ s+b[i]= sum(b[k]) k=0,i-10 ≤ 1 < 11 ^ s+b[i]= sum(b[k]) k=0,i0 ≤ 1 < 11 ^ s+b[i]= sum(b[k]) k=0,i-1+b[i]0 ≤ 1 < 11 ^ s+b[i]= sum(b[k]) k=0,i-1+b[i]0 ≤ 1 < 11 ^ s = sum(b[k]) k=0,i-1
3. Post condition: (when i=11)I : 1 ≤ i ≤ 11 ^ s = sum(b[k]) k=0, ..., i-1
1 ≤ 11 ≤ 11 ^ s = sum(b[k]) k=0, ..., 10 True ^ True
)61(
Static Program Testing Analysis:2. Design by Contract (DBC)
Introduction:- Software entities have obligations to other entities based upon formalized rules between them.- A functional specification, or 'contract', is created for each module in the system before it is
coded. Program execution is then viewed as the interaction between the various modules as bound by these contracts.
- routines have explicit preconditions that the caller must satisfy before calling the routine, and explicit postconditions that describe the conditions that the routine will guarantee to be true after the routine finishes
- a contract takes the following general form: "If you, the caller, set up certain preconditions, then I will establish certain other results when I return to you. If you violate the preconditions, then I promise nothing."
- DBC is novel in recognizing that these contracts are so crucial to software correctness that they should be part of the design process. In effect, DBC advocates writing the assertions first.
- The notion of a contract extends down to the method/procedure level; the contract for each method will normally contain the following pieces of information:
o Acceptable and unacceptable inputs o Return values, and their meanings o Error and exception conditions that can occur o Side-effects o Preconditions o Postconditions o invariants o (Rarer) Performance guarantees, e.g., for time or space used
DBC Syntax:- When you write a module, as part of this program, you have to follow the syntax:
Requirement---------
Programs------
Ensure------
- If a requirement is met, then the ensure is generated to be generated.
)62(
Glossary
Taken from ADRION, W. R., BRANSTAD, M. A. & CHERNIAVSKY, J. C. (1982) Validation, Verification, and Testing of Computer Software. ACM Comput. Surv., 14, 159-192.
Audit.See DOD Development Reviews.
Black Box Testing.See Functional Testing.
Boundary Value Analyses. A selection technique in which test data are chosen to lie along "boundaries" of input domain (or output range) classes, data structures, procedure parameters, etc. Choices often include maximum, minimum, and trivial values or parameters. This technique is often called stress testing. (See Section 2.4.)
Branch Testing. A test method satisfying coverage criteria that require that for each decision point each possible branch be executed at least once. (See Section 2.5.)
Cause-Effect Graphing. Test data selection technique. The input and output domains are partitioned into classes andanalysis is performed to determine which input classes cause which effect. A minimal set of inputs is chosen that will cover the entire effect set. (See Section 2.4.)
Certification.Acceptance of software by an authorized agent usually after the software has been validated by the agent, or after its validity has been demonstrated to the agent.
Critical Design Review. See DOD Development Reviews.
Complete Test Set. A test set containing data that causes each element of a prespecified set of Boolean conditions to be true. Additionally, each element of the test set causes at least one condition to be true. (See Section 2.2.)
Consistent Condition Set. A set of Boollean conditions such that complete test sets for the conditions uncover the same errors. (See Section 2.2.)
Cyclomatic Complexity. The cyclomatic complexity of a program is equivalent to the number of decision statements plus1. (See Section 2.5.)
DD (decision-to-decision) Path. A path of logical code sequence that begins at an entry or decision statement and ends at a decision statement or exit. (See Section 2.5.)
Debugging. The process of correcting syntactic and logical errors detected during coding. With the primary goal of obtaining an executing piece of code, debugging shares with testing certain techniques and strategies, but differs in its usual ad hoc application and local scope.
Design-Based Functional Testing.The application of test data derived through functional analysis (see Functional Testing) extended to include design functions as well as requirement functions. (See Section 2.4.)
DOD Development Reviews.
)63(
A series of reviews required by DOD directives. These include(1) The Systems Requirements Review is an examination of the initial progress during the problem
definition stage and of the convergence on a complete system configuration. Test planning and test documentation are begun at this review.
(2) The System Design Review occurs when the system definition has reached a point where major system modules can be identified and completely specified along with the corresponding test requirements. The requirements for each major subsystem are examined along with the preliminary test plans. Tools required for verification support are identified and specified at this stage.
(3) The Preliminary Design Review is a formal technical review of the basic design approach for each major subsystem or module. The revised requirements and preliminary design specifications for each major subsystem and all test plans, procedures, and documentation are reviewed at this stage. Development and verification tools are further identified at this stage. Changes in requirements will lead to an examination of the test requirements to maintain consistency.
(4) The Critical Design Review occurs just prior to the beginning of the construction stage. The complete and detailed design specifications for each module and all draft test plans and documentation are examined. Again, consistency with previous stages is reviewed, with particular attention given to determining if test plans and documentation reflect changes in the design specifications at all levels.
(5) Two audits, the Functional Configuration Audit and the Physical Configuration Audit are performed. The former determines if the subsystem performance meets the requirements. The latter audit is an examination of the actual code. In both audits, detailed attention is given to the documentation, manuals and other supporting material.
(6) A Formal Qualification Review is performed to determine through testing that the final coded subsystem conforms with the final system specifications and requirements. It is essentially the subsystem acceptance test.
Driver. Code that sets up an environment and calls a module for test. (See Section 1.3.)
Dynamic Analysis.Analysis that is performed by executing the program code. (See Section 2.7.)
Dynamic Assertion. A dynamic analysis technique that inserts assertions about the relationship between program variables into the program code. The truth of the assertions is determined as the program executes. (See Section 2.7.)
Error Guessing. Test data selection technique. The selection criterion is to pick values that seem likely to cause errors. (See Section 2.4.}
Exhaustive Testing. Executing the program with all possible combinations of values for program variables. (See Section 2.1.)
Extrenal Test Data. Test data that is at the extreme or boundary of the domain of an input variable or which produces results at the boundary of an output domain. (See Section 2.4.)
Formal Qualification Review. See DOD Development Reviews.
Functional Configuration Audit. See DOD Development Reviews.
Functional Testing. Application of test data derived from the specified functional requirements without regard to the final program structure. (See Section 2.4.)
Infeasible Path. A sequence of program statements that can never be executed. (See Section 2.5.)
Inspection.
)64(
A manual analysis technique in which the program {requirements, design, or code) is examined in a very formal and disciplined manner to discover errors. (See Section 2.2.)
Instrumentation. The insertion of additional code into the program in order to collect information about program behavior during program execution. (See Section 2.7.)
Invalid Input (Test Data for Invalid Input Domain). Test data that lie outside the domain of the function the program represents. (See Section 2.1.)
Life-Cycle Testing. The process of verifying the consistency, completeness, and correctness of the software entity at each stage in the development. (See Section 1.)
Metric-Based Test Data Generation.The process of generating test sets for structural testing based upon use of complexity metrics or coverage metrics. (See Section 2.5.)
Mutation Analysis. A method to determine test set thoroughness by measuring the extent to which a test set can discriminate the program from slight variants (mutants) of the program. (See Section 2.6.)
Oracle. A mechanism to produce the "correct" responses to compare with the actual responses of the software under test. (See Section 2.1.)
Path Expressions. A sequence of edges from the program graph which represents a path through a program. (See Section 2.5.)
Path Testing. A test method satisfying coverage criteria that each logical path through the program be tested. Often paths through the program are grouped into a Testing of Computer Software • 187 finite set of classes; one path from each class is then tested. (See Section 2.5.)
Preliminary Design Review. See DOD Development Reviews.
Program Graph. Graphical representation of a program. (See Section 2.5.)
Proof of Correctness. The use of techniques of mathematical logic to infer that a relation between program variables assumed true at program entry implies that another relation between program variables holds at program exit. (See Section 2.2.)
Regression Testing. Testing of a previously verified program required following program modification for extension or correction. (See Section 1.4.)
Simulation. Use of an executable model to represent the behavior of an object. During testing the computational hardware, the external environment, and even code segments may be simulated. (See Section 2.2.)
Self-Validating Code. Code which makes an explicit attempt to determine its own correctness and to proceed accordingly. (See Section 2.7.)
Special Test Data.
)65(
Test data based on input values that are likely to require special handling by the program. (See Section 2.4.)
Statement Testing. A test method satisfying the coverage criterion that each statement in a program be executed at least once during program testing. (See Section 2.5.)
Static Analysis,Analysis of an program that is performed without executing the program. (See Section 2.7.)
Stress Testing. See Boundary Value Analysis.
Structural Testing. A testing method where the test data are derived solely from the program structure. (See Section 2.5.)
Stub. Special code segments that, when invoked by a code segment under test, will simulate the behavior of designed and specified modules not yet constructed. (See Section 1.3.)
Symbolic Execution. A static analysis technique that derives a symbolic expression for each program path. (See Section 2.7.)
System Requirements Review. See DOD Development Reviews.
Test Data Set. Set of input elements used in the testing process. (See Section 2.1.)
Test Driver. A program that directs the execution of another program against a collection of test data sets. Usually the test driver also records and organizes the output generated as the tests are run. (See Section 1.3.)
Test Harness. See Test Driver.
Testing. Examination of the behavior of a program by executing the program on sample data sets.
Valid Input (test data for a valid input domain). Test data that lie within the domain of the function represented by the program. (See Section 2.1.)
Validation. Determination of the correctness of the final program or software produced from a development project with respect to the user needs and requirements. Validation is usually accomplished by verifying each stage of the software development life cycle.
Verification. In general, the demonstration of consistency, completeness, and correctnessof the software at each stage and between each stage of the development life cycle.
Walk-Through. A manual analysis technique in which the module author describes the module's structure and logic to an audience of colleagues. (See Section 2.2.)
White Box Testing. See Structural Testing.
)66(