Software Testing & Quality Assurance

Overview

� The Psychology of Testing

� Testing terminology and techniques

– i.e., black-box vs. white-box testing, static vs. dynamic

� Integration Testing

� Software Inspections

� Control-Flow Analysis

TESTING 1

So you think you know testing?...

Consider the following example:

Write a suite of tests for a program that reads three integers from the

command line. Those three values will be interpreted as the lengths of

the legs of a triangle. The program will determine if this triangle is

equilateral, isoceles, or scalene and output its result.

� Write down a set of inputs which you think would be good tests for this

program.

TESTING 2

The dark side of testing

� The majority of start-up companies (and some established ones) today have a

rapid development mentality.

� They tend to employ quick and dirty schemes of developing software.

� Unfortunately, today’s quick and dirty rapid development schemes have

major holes in them.

� “Yes, we ran it and it worked”

� “Yes, I sat down and typed lots of inputs at it and it didn’t fall over.”

� “No, we don’t have time. It compiled and linked OK.”

TESTING 3

Problems with these naïve views

� “Yes, we ran it and it worked”

� So what requirements did you test?

� “Yes, I sat down and typed lots of inputs at it and it didn’t fall over.”

� So what part of the system passed or failed?

� “No, we don’t have time. It compiled and linked OK.”

� So when can testing start?

We need systematic approaches to finding out if our software runs properly.

i.e., according to what we intended when we set out to build our software.

TESTING 4

The Psychology of Testing

“Testing is the process of executing a program with the intent of finding

errors.” Glenford Myers, The Art of Software Testing

� Because we are human, if we are supposed to prove the correctness of a

program, we will undeliberately run tests that are less likely to detect faults.

� Software developers have this mentality.

� “If the goal is to show the absence of errors, we will find very few of

them.”

� However, if the goal of a test is to find faults, then we will automatically

orient our efforts in this direction.

� Software testers have this mentality.

� “If the goal is to show the presence of errors, we will discover a large

number of them.”

TESTING 5

Testing vs. Debugging

These are two distinct processes related to the psychology of developers and

testers.

� Verification and validation (i.e., testing ) is concerned with establishing the

existence of defects in a program.

� This is along the psyche of the software testers.

� Debugging is concerned with locating and repairing these defects.

� It also involves formulating a hypothesis about program behaviour and

testing this hypothesis to find the error.

� This is along the psyche of the software developers.

TESTING 6

Verification and Validation

Validation

� Is the right thing built?

� The software should do what the customer really requires.

i.e., was the specification what the customer intended?

Verification

� Is the thing built right?

� The software must conform to its specification.

We will concentrate most on verification techniques, as validation deals more

with requirements analysis.

TESTING 7

Static vs. Dynamic Verification

These are two methods used to verify programs.

Software testing (dynamic)

� This method is concerned with executing the software product using test data

and observing its behaviour.

Software inspections (static)

� This method is concerned with the analysis of the static system

representations to discover faults.

� It may be supplemented by tool-based analysis of documents and program

code.

TESTING 8

Testing Terminology

Let’s revisit the different error definitions.

� A fault is the design flaw, the mistake someone made that eventually caused

the system to fail (or not).

� This is a static phenomenon.

� A failure is the event that made you take notice that something is wrong.

� It’s the crash, the bad output.

� It may occur far from the fault.

� This is a dynamic phenomenon.

TESTING 9

Testing Terminology (ctd.)

Some more jargon that you should be familiar with:

� A test case is the collection of inputs, expected results, environment and

procedural requirement for a single test.

� A test suite is a collection of test cases necessary to “adequately” test a

product.

� A test plan is a document describing the scope, approach, resources, and

scheduled testing activities.

� A test harness (a.k.a test driver) is some code driver that is used to

(interactively) test the routine(s)/classes required.

TESTING 10

Classification of Types of Testing

There are different types of testing used at various stages of the testing process.

� Structured testing (i.e., white-box testing) are test cases generated based on

the internal structure of the software.

– i.e., Structural testing � white-box techniques.

� Functional testing (i.e., black-box testing) are test cases generated based on

the software specification.

– i.e., Functional testing � black-box techniques.

� Grey-box testing is based on test cases for class or module interactions.

– Slightly more recent idea for OO and structurally rich designs.

– Not dependent on code details but more than just call-and-return

functional testing.

– The tests must be established after detailed software design.

TESTING 11

Testing stages vs. Testing types

There are five stages of testing usually found in the development of a system:

1. Unit

2. Integration

3. System

4. Installation

5. Acceptance

Unit Integration System Installation Acceptance

Structural Y ? N N N

Functional N ? Y Y Y

Grey-box N Y N N N

TESTING 12

Structural (White-box) Testing

� Here, we get to look in to the source code as much as possible.

– i.e., we examine code-level issues such as statements, blocks, logical

paths

– The test cases are volatile as they may not survive maintenance.

� There are two forms of white-box testing which we will examine:

1. Logic Coverage Testing

2. Loop Testing

TESTING 13

What is Logic Coverage Testing?

� Logic Coverage is one of the oldest and most widely used white-box testing

strategies today.

� Test cases are derived from the program’s logical structure.

� It attempts to expose software defects by exercising a unique combination of

the program’s statements known as a path.

– Logic coverage is sometimes referred to as path testing.

TESTING 14

Logic Coverage Testing Strategies

� There are different strategies within logic coverage testing.

� They vary in their coverage criteria.

– This is the domain of paths through the program that acheive a

“complete” test.

1. Exhaustive Path Coverage Criterion

2. Statement Coverage Criterion

3. Decision (or Branch) Coverage Criterion

4. Condition Coverage Criterion

5. Decision/Condition Coverage Criterion

6. Multiple Condition Coverage Criterion

TESTING 15

Exhaustive Path Testing

� Theoretically the most complete of the logic coverage criteria.

� It requires EVERY possible path to be exercised.

� Unfortunately, even programs of modest complexity can have an enormous

number of paths due to the combinatorial explosion of loops and branches.

– In reality, this is almost impossible to achieve.

TESTING 16

Statement Coverage Testing

� This is the simplest coverage criterion with the fewest number of test cases.

� It requires every statement in the program to be executed at least oncea.

� Consider the following piece of code:

if (A && B) {

C;

}

D;

� A single test case (A is true and B is true) will satisfy the criterion.

� We haven’t verified that statement C will NOT be executed when A is false

and/or B is false.

� Statement coverage is a necessary but not sufficient criterion for effectivestructural testing.

aUntil we switched to Java for first and second year courses, there was a tool calledanalyze coverage which ran these types of coverage tests for Modula-3 programs.

TESTING 17

Decision (or Branch) Coverage Testing

� Decision coverage requires that each branch within the decision be evaluated.

� When considering the code from the previous slide, only two test cases

would be required to satisfy this criterion.

– A is true and B is truea

– A is false and B is true

� Two other cases derived from the condition are left unevaluated.

– A is true and B is true

– A is false and B is false

� Decision coverage subsumes statement coverage.

� However, decision coverage does not detect all faults in compound

conditions.

aAlso satisfies the statement coverage criterion

TESTING 18

Condition Coverage Testing

� This is similar to decision coverage but it is more sensitive to the control flow.

� Condition coverage requires that each condition in a decision to take on all

possible values at least once.

� Consider a slightly different piece of code:

if (A || B) {

C;

}

D;

� So, only two test cases would be required to satisfy this criterion.

– A is true and B is falsea

– A is false and B is truea

� However, these test cases neglect to bypass statement C.

� Thus, the decision coverage criterion is no longer satisfied.aAlso satisfies the statement coverage criterion

TESTING 19

Decision/Condition Coverage Testing

� This combines both the decision and condition coverage criteria.

� It requires that each condition take on all possible outcomes at least once and

each decision take on all possible outcomes at least once.

� Again from the previous slide,

if (A || B) {

C;

}

D;

� But, still we can find just two test cases required to satisfy this criterion.

– A is true and B is truea

– A is false and B is false

aAlso satisfies the statement coverage criterion

TESTING 20

Multiple-Condition Coverage Testing

� Multiple-condition coverage subsumes statement coverage, decision

coverage, condition coverage and decision/condition coverage criteria and

does not mask conditions.

� i.e., In the previous example, (A is false and B is true) was masked

because it was subsumed in the first test case.

� It requires all possible combinations of condition outcomes in each decision.

� Thus, there are

� n combinations where n is the number of conditions.

� Unfortunately, it is extremely tedious to determine the minimum set of test

cases required, especially for complex Boolean expressions.

TESTING 21

Loop Testing

� There are four types of loops that need to be examined in structural testing:

1. Deterministic Loops (usually for)

� The number of times the loop will execute is known in advance.

� Before the first statement is executed, a variable is defined that will

not change throughout the running of the loop.

2. Non-deterministic Loops (usually while)

� The number of times the loop will execute is not known in advance.

� There are three instances where non-deterministic loops may arise:

(a) The max. value is unknown.

(b) Processing within the loop changes the control variable, if any.

(c) A condition contained in the loop causes it to exit hastily.

3. Nested Loops

4. Unstructured Loops

� Occurs when a process jumps out of or into the middle of a loop.

TESTING 22

Loop Testing: Critical Test Values

� Critical test values are a combination of values allowing loops to proceed

through normal and faulty operation.

Bypass: any value that will cause the loop to exited immediately.

Once: values causing the loop to be executed once.

Twice: values causing the loop to be executed exactly twice.

Typical: a typical number of iterations.

Max: the maximum number of allowed iterations.

Max + 1: one more than the max allowed.

Max - 1: one less than the max allowed.

Min: the minimum required.

Min - 1: one less than the minimum required.

Null: no input.

Negative: negative input.

TESTING 23

Loop Testing Example: Course Handout System

� We are testing a software system that handles course handouts with a

minimum of 1 and a maximum of 12 students in the class.

� Using critical test value criteria, we design the following inputs:

Bypass: No students.

Once: One student.

Twice: Two students.

Typical: 10 students.

Max: 12 students.

Max + 1: 13 students.

Max - 1: 11 students.

Min: Redundant.

Min - 1: Redundant

Null: Redundant

Negative: Negative # of students?

� After defining our critical test values, we would test the looping specification

with the inputs and examine the theoretical output.

� Remember: even though some cases above are a stretch, testers are paidto cause software behaviour to act out of the ordinary.

TESTING 24

Loop Testing: Conclusions

� Loop testing is a common technique based on the experience programmers

have with errors in starting and finishing loops.

� i.e., the majority of loops aren’t simple for-loops.

� The most common types of loops implemented are the deterministic,

non-deterministic and nested loops.

� By using the method of defining critical test values, an analyst uses systems

specifications to design tests capable of breaking certain loops prematurely.

� This technique is also effective for testing not only control-flow graphs,

but also all flow models with the possible exception of finite state

machines (too many loops).

TESTING 25

Integration Testing

� This method of testing is needed whenever modules or subsystems are

combined to create a larger system.

� The goal is to identify errors due to interface faults or to invalid interface

assumptions.

� This technique is particularly important in OO system developement.

TESTING 26

Integration Testing: Types of Interfaces

� Shared Memory Interfaces (Concurrent packages)

� A block of memory is shared between components.

� Procedural Interfaces (e.g., libraries, components)

� Set of procedures/classes encapsulated in a package or a subsystem.

� Message-Passing Interfaces (client-server)

� Subsystems require services from each other.

� Pass the messages from the client to the server.

TESTING 27

Integration Testing: Interface Errors

� Interface misuse

� Wrong parameter order.

� Wrong number of parameters.

� Wrong parameter types.

� Interface misunderstanding

� The call to the component make incorrect assumptions about the

component being called.

� Timing errors (threads and concurrency)

� Race conditions.

� Data synchronization errors.

TESTING 28

Integration Testing Guidelines

� Design your tests so actual parameters passed are at extreme ends of formal

parameter ranges.

� i.e., Percentile ranges should be tested at 0 and 100.

� Test pointer variables with null values.

� Frequently, the problem with objects being passed is that they are not

initialized.

� Design tests that cause components to fail.

� Use stress testing in message passing systems.

� i.e., make sure that the server can handle the load from the number of

clients.

� In shared memory systems, vary the order in which components are activated.

� This can frequently lead to a combinatorial explosion of possible

combinations and should be first examined more closely.

TESTING 29

System Testing Guidelines

� Testing of critical systems must often rely on simulators for sensor and

activator data.

� You don’t want to endanger people and/or profit.

� Testing for normal operations should be done using a safely obtained

operational profile.

� Tests for exceptional conditions will need to involve simulators.

TESTING 30

Software Inspections and Walk-throughs

� A software inspection is an example of qualified individuals formally

examining a source code representation to discover anomalies and defects.

� An inspection does not require the system to execute, so it may occur

before implementation.

� It may be applied to any type of system documentation.

– i.e., document, model, test data, code, etc.

� Walk-throughs are informal inspections.

� Inspections are very effective for discovering defects.

� It is possible to discover several defects in a single inspection.

� The inspections use the domain and programming knowledge of the

“experts” reviewing the code.

– i.e., reviewers help the authors avoid make common errors.

TESTING 31

Inspections and Testing

� Inspections and testing are complementary processes.

� Inspections are used to check conformance to specifications.

� Testing checks compliance with non-functional system characteristics

such as performance, usability, etc...

� Like testing, inspections focus on defect detection, not defect correction.

� Defects uncovered may be logic errors, coding errors, or non-compliance

with development standards.

� Inspections and walk-throughs are considered two of the first methods of

testing without the use of a computer.

� These methods have also been called human testing.

TESTING 32

Before you start your inspection...

� A precise specification must be available.

� The reviewing team members must be familiar with organization standards.

� All representations must be syntactically correct.

� An error checklist must be prepared in advance.

� Management must buy into the fact that inspections will increase the early

development costs.

� Inspections cannot be used to evaluate staff performance.

TESTING 33

Doing the inspection...

1. Code and associated documents are distributed to the team in advance.

2. The system overview is presented to the inspection team and the inspection is

driven via a checklist of common errors.

� The error checklist should be language dependent.

� The weaker the type-checking of the language, the larger the checklist is

likely to become.

3. Errors discovered during the inspection are recorded.

4. Product modifications are recorded to repair defects.

� Re-inspection may or may not be required.

TESTING 34

The Inspection Team

There are at least four members on the inspection team:

� The product author

� The inspector

– Looks for errors, omissions, and inconsistencies.

� The reader

– Reads the code to the team.

� The moderator

– Chairs meeting and records errors uncovered.

TESTING 35

Inspection Fault Classes

� There are several types of fault classes to observe when performing an

inspection.

– Data faults (e.g., array bounds, variables not initialized)

– Control faults (e.g., loop termination, unreachable code)

– Input/output faults (e.g., all the data read, duplicate variables output)

– Interface faults (e.g., parameter assignment & type mismatches)

– Storage management faults (e.g., memory leaks, pointer arithmetic)

– Exception management faults (e.g., all error conditions trapped)

TESTING 36

Spending Time on Inspections

� There are three places where the system is reviewed.

1. During the overview

– approx. 500 statements per hour.

2. During individual preparation

– approx. 125 statements per hour.

3. During the team team inspection

– approx. 90-125 statements per hour.

� Including preparation time, each 100 lines of code costs approximately

one-person day, if a 4 person team is used.

TESTING 37

Automated Static Analysis Tools vs. Inspections

� Performed by software tools that process source code listings.

– i.e., integrated tools into commercial IDEs, program understanding tools,

reverse engineering tools, etc.

� They can be used to flag potentially erroneous conditions for the inspection

team to examine.

� These tools should supplement before hand the reviews done by the

inspectors.

TESTING 38

Control Flow Testing

� The control flow of a program may be modelled by a directed graph.

� Nodes can be considered as statements functions, procedures, objects, or

even subsystems.

– Nodes that are fundamental indivisible operations or statements are

called basic blocks.

� Edges are directed relations between the nodes.

– i.e., interface connections (procedural, message-passing ...)

� These flows can model representations of sequences, selections and iterations

within a set of specifications.

� Control flow graphs are used in static analysis.

� This technique is not restricted to structural (white-box) or functional

(black-box) testing.

– However, we will focus on its capabilities for black-box testing of

specifications.

TESTING 39

Control Flow Testing Definitions

� An independent path is any path that

introduces at least one new set of

nodes.

– i.e., it must traverse an edge not

previously covered in another

path.

What are the independent paths in

this example?6

1

2

3

5

4

� The cyclomatic complexity is the upper bound on the number of independent

paths.

cyclomatic complexity = edges - nodes + 1

TESTING 40

Control Flow Testing Definitions (ctd.)

� An predicate node is a node that

has two or more out-edges, such

that each edge is weighted by a

corresponding predicate value

(likely a logical boolean result).

� A selector node is a node with

more than two out-edges, where

each edge is weighted by a

selector value

� A junction node is a node with

two or more in-edges.

11

11.1

11.4

11.3

11.2

9

8’8TRUE

FALSE

Married

Divorced

Widow

Single

TESTING 41

Modeling Compound Predicates

� Analyzing and testing compound predicates can be daunting due to the level

of complexity hidden within them.

� The higher the complexity � the higher the likelihood that a softwareengineer has introduced an error during coding.

� So, to model a compound predicate, you would build a predicate tree with as

many branches as there are possibilities.

� The rule is for n predicates, there are

� n branches.

� Building a predicate tree:

1. Break the compound predicate into separate nodes.

2. The branches are then attached to the model and flow of the predicate is

then determined.

� To validate a compound predicate, you must model

� n cases independent ofone another

� Alternative: Build a truth table.

TESTING 42

Design and Execution Steps in Control Flow Testing

1. Examine the requirements and analyze them for operationally satisfactory

completeness.

2. Rewrite the specification as a sequence of short sentences.

3. Number the sentences uniquely (i.e., 1, 2, 3, ...)

4. Build the model.

5. Verify the model because your work is as “buggy” as the programmer’s code.

6. Select test paths.

7. Sensitize the paths you picked.

8. Predict and record the expected outcome for each test.

9. Define the validation criteria for each test.

10. Run the tests.

11. Confirm the outcomes.

12. Confirm the path.

TESTING 43

Control Flow Testing Example - A Vending Machine

� Examine the Insert Coin function for a vending machine program.

– Requirements:

1. The coin is inserted.

2. Check if the coin is legal.

3. If the coin is illegal, then throw an exception.

4. If the coin is legal, then add it to the current value inserted.

5. Display the total coins inserted on screen.

� Even though we don’t have the code for this function, we can still apply

control-flow testing as a black-box approach.

� Let’s follow the 12 design and execution steps to model this function.

TESTING 44

Control Flow Testing Example - A Vending Machine

1. Label each line of the insertmethod independently which will

be used as predicate labels.

2. Once we have the code labeled,we build the control-flow model.

3. Since we only have one predicate

node, node 2, in this model, thenext step of selecting our test

paths is rather simple.

4. Next we must sensitize the twotest paths by selecting input values

that would cause the method to do

the equivalent traversal in theabsence of errors.

5. After defining the inputs, we

researched similar test cases and

predicted the outcome of our tests.

6. Now we use the inputs to run the

model and record the outputs.

1

2

3 4-5

Exception Exit

TESTING 45

Control Flow Testing Conclusions

� Control-flow testing is a fundamental model of functional testing.

� To build a control-flow model, you must refer to documents such as the

Software Requirements Specification to ensure cohesiveness between the test

and the specification.

� During path design, you must ensure that all of the links are covered by as

many test paths as needed.

� In the end, the validity of a test case can only be ensured if the program and

the test model are bug free.

TESTING 46

Software Testing Conclusions

� There are many ways to approach software testing.

� However, the best frame of mind to be in is to search for errors in a

program, and not to check and see if the program works properly.

� This methodology can be done statically via code inspections, or

dynamically by verifying the code’s behaviour using test cases.

� There are two fundamental methods used to approach dynamic testing:

1. Structural (white-box) testing, which bases test-cases on the internal

structure of the software (e.g., coverage & loop testing).

2. Functional (black-box) testing, which bases test-cases on the software’s

specifications (e.g., control-flow).

� There are different stages of testing for different views of the system.

– i.e., unit, integration, system, acceptance, etc.

TESTING 47