slide 13.1 copyright © 2004 by the mcgraw-hill companies, inc. all rights reserved. an introduction...

1

Copyright © 2004 by The McGraw-Hill Companies, Inc. All rights reserved.

An Introduction toObject-Oriented

Systems Analysis and Design with UML and

the Unified Process

McGraw-Hill, 2004

Stephen R. Schach

[email protected]

2


CHAPTER 13

TESTING

3


Chapter Overview

Introduction to Testing Quality Issues Nonexecution-Based Testing Execution-Based Testing The Two Basic Types of Test Cases What Execution-Based Testing Should Test Who Should Perform Execution-Based Testing? When Testing Stops

4


Introduction to Testing

Traditional life-cycle models usually include a separate testing phase, after implementation and before maintenance– This cannot lead to high-quality information systems

Testing is an integral component of the information system development process– An activity that must be carried out throughout the life

cycle

5


Introduction to Testing (contd)

It is insufficient to test the artifacts of a workflow merely at the end of that workflow

Continual testing carried out by the development team while it performs each workflow is essential,– In addition to more methodical testing at the end of each

workflow

6



Verification – The process of determining whether a specific workflow

has been correctly carried out– This takes place at the end of each workflow

Validation– The intensive evaluation process that takes place just

before the information system is delivered to the client– Its purpose is to determine whether the information system

as a whole satisfies its specifications

The term V & V is often used to denote testing

7



The words verification and validation are used as little as possible in this book– The phrase verification and validation (or V & V) implies

that the process of checking a workflow can wait until the end of that workflow

– On the contrary, this checking must be carried out in parallel with all information system development and maintenance activities

To avoid the undesirable implications of the phrase V & V, the term testing is used instead– This terminology is consistent with the Unified Process,

which uses the term “test workflow”

8



There are two types of testing– Execution-based testing of an artifact means running

(“executing”) the artifact on a computer and checking the output

However, a written specification, for example, cannot be run on a computer– The only way to check it is to read through it as carefully as

possible– This type of checking is termed nonexecution-based testing

» (Unfortunately, the term verification is sometimes also used to mean nonexecution-based testing. This can also cause confusion.)

9



Clearly, computer code can be tested both ways– It can be executed on a computer, or– It can be carefully reviewed

Reviewing code is at least as good a method of testing code as executing it on a computer

10


Quality Issues

The quality of an information system is the extent to which it satisfies its specifications

The term quality does not imply “excellence” in the information systems context – Excellence is generally an order of magnitude more than

what is possible with our technology today

11


Quality Issues (contd)

The task of every information technology professional is to ensure a high-quality information system at all times– However, the information system quality assurance group

has additional responsibilities with regard to information system quality

12


Quality Assurance Terminology (contd)

A fault is the standard IEEE terminology for what is popularly called a “bug”

A failure is the observed incorrect behavior of the information system as a consequence of the fault

An error is the mistake made by the programmer

In other words,– A programmer makes an error that results in a fault in the

information system that is observed as a failure

13


Managerial Independence (contd)

It is important to have managerial independence between – The development team

and– The quality assurance group

14



Serious faults are often found in an information system as the delivery deadline approaches– The information system can be released on time but full of

faults» The client then struggles with a faulty information system

or – The developers can fix the information system but deliver

it late

Either way the client will lose confidence in the information system development organization

15



A senior manager should decide when to deliver the information system – The manager responsible for development, and – The quality assurance manager – Should report to the more senior manager

The senior manager can decide which of the two choices would be in the best interest of both the development organization and the client

16



A separate quality assurance group appears to add greatly to the cost of information system development– The additional cost is one manager to lead the quality

assurance group

The advantage is a quality assurance group consisting of independent specialists

In a development organization with under six employees – Ensure that each artifact is checked by someone other than

the person responsible for producing that artifact

17


Nonexecution-Based Testing

When we give a document we have prepared to someone else to check– He or she immediately finds a mistake that we did not see

It is therefore a bad idea if the person who draws up a document is the only one who reviews it– The review task must be assigned to someone other than

the author of the document– Better still, it should be assigned to a team

18


Nonexecution-Based Testing (contd)

This is the principle underlying the inspection– A review technique used to check artifacts of all kinds – In this form of nonexecution-based testing, an artifact is

carefully checked by a team of information technology professionals with a broad range of skills

19


Nonexecution-Based Testing (contd)

Advantages:

The different skills of the participants increase the chances of finding a fault

A team often generates a synergistic effect– When people work together as a team, the result is often

more effective than if the team members work independently as individuals

20


Principles of Inspections (contd)

An inspection team should consist of from 4 to 6 individuals– Example: An analysis workflow inspection team

» At least one systems analyst» The manager of the analysis team» A representative of the next team (design team)» A client representative» A representative of the quality assurance group

An inspection team should be chaired by the quality assurance representative – He or she has the most to lose if the inspection is

performed poorly and faults slip through

21



The inspection leader guides the other members of the team through the artifact to uncover any faults– The team does not correct faults– It records them for later correction

22



There are four reasons for this:– A correction produced by a committee is likely to be lower

in quality than a correction produced by a specialist

– A correction produced by an team of (say) five individuals will take at least as much time as a correction produced by one person and, therefore, costs five times as much

– Not all items flagged as faults actually are incorrect» It is better for possible faults to be examined carefully at a later

time and then corrected only if there really is a problem

– There is not enough time to both detect and correct faults» No inspection should last longer than 2 hours

23



During an inspection, a person responsible for the artifact walks the participants through that artifact – Reviewers interrupt when they think they detect a fault– However, the majority of faults at an inspection are

spontaneously detected by the presenter

24



The primary task of the inspection leader is – To encourage questions about the artifact being inspected

and promote discussion

It is absolutely essential that the inspection not be used as a means of evaluating the participants

If that happens– The inspection degenerates into a point-scoring session– Faults are not detected

25



The sole aim of an inspection is to highlight faults – Performance evaluations of participants should not be

based on the quality of the artifact being inspected– If this happens, the participant will try to prevent any faults

coming to light

26



The manager who is responsible for the artifact being reviewed should be a member of the inspection team– This manager should not be responsible for evaluating

members of the inspection team (and particularly the presenter)

– If this happens, the fault detection capabilities of the team will be fatally weakened

27


How Inspections are Performed

An inspection consists of five steps:

An overview of the artifact to be inspected is given– Then the artifact is distributed to the team members

In the second step, preparation, the participants try to understand the artifact in detail– Lists of fault types found in recent inspections ranked by

frequency help team members concentrate on areas where the most faults have occurred

28


How Inspections are Performed

In the inspection, one participant walks through the artifact with the inspection team– Fault finding now commences– The purpose is to find and document faults, not to correct

them– Within one day the leader of the inspection team (the

moderator) produces a written report of the inspection

The fourth stage is the rework– The individual responsible for that artifact resolves all

faults and problems noted in the written report

29


How Inspections are Performed (contd)

In the follow-up, the moderator ensures that every single issue raised has been resolved satisfactorily– By either fixing the artifact or – Clarifying items incorrectly flagged as faults– If more than 5 percent of the material inspected has been

reworked, the team must reconvene for a 100 percent reinspection

30


How Inspections are Performed (contd)

Input to the inspection:– The checklist of potential faults for artifacts of that type

Output from the inspection– The record of fault statistics

» Recorded by severity (major or minor), and » Fault type

The fault statistics can be used in a number of different ways

31


Use of Fault Statistics

The number of faults observed can be compared with averages of faults detected in those same artifact types in comparable information systems– This gives management an early warning that something

is wrong, and– Allows timely corrective action to be taken

If a disproportionate number of faults of one type are observed, management can take corrective action

32


Use of Fault Statistics (contd)

If the detailed design of a module reveals far more faults than in any other module– That module should be redesigned from scratch

Information regarding the number and types of faults detected at a detailed design inspection will aid the team performing the code inspection of that module at a later stage

33



The results of experiments on inspections have been overwhelmingly positive

Typically, 75 percent or more of all the faults detected over the lifetime of an information system are detected during inspections before execution-based testing of the modules is started

Savings of up to $25,000 per inspection have been reported

Inspections lead to early detection of faults

34


Execution-Based Testing

Requirements workflow

Analysis workflow

Design workflow– The artifacts of these workflow are diagrams and

documents– Testing has to be nonexecution-based

Why then do systems analysts need to know about execution-based testing?

35


The Relevance of Execution-Based Testing

Not all information systems are developed from scratch

The client’s needs may be met at lower cost by a COTS package

In order to provide the client with adequate information about a COTS package, – The systems analyst has to know about execution-based

testing

36


Principles of Execution-Based Testing

Claim:– Testing is a demonstration that faults are not present

Fact:– Execution-based testing can be used to show the

presence of faults– It can never be used to show the absence of faults

37


Principles of Execution-Based Testing (contd)

Run an information system with a specific set of test data– If the output is wrong then the information system

definitely contains a fault, but– If the output is correct, then there still may be a fault in the

information system» All that is known from that test is that the information system runs

correctly on that specific set of test data

38


Principles of Execution-Based Testing (contd)

If test data are chosen cleverly– Faults will be highlighted

If test data are chosen poorly– Nothing will be learned about the information system

39


The Two Basic Types of Test Cases

Black-box test cases:– Drawn up by looking at only the specifications

» The code is treated as a “black box” (in the engineering sense)

Glass-box test cases– Drawn up by carefully examining the code and finding a

set of test cases that, when executed, will together ensure that every line of code is executed at least once» These are called glass-box test cases because now we look inside

the “box” and examine the code itself to draw up the test cases

40


What Execution-Based Testing Should Test

Correctness is by no means enough

Four other qualities need to be tested: – Utility– Reliability– Robustness– Performance

41



Utility is the measure of the extent to which an information system meets the user’s needs– Is it easy to use?– Does it perform useful functions?– Is it cost effective?

42



Reliability is a measure of the frequency and criticality of information system failure– How often does the information system fail?

» (Mean time between failures)

– How bad are the effects of that failure?

– How long does it take to repair the system?» (Mean time to repair)

– How long does it take to repair the results of the failure?

43



Robustness is a measure of a number of factors including– The range of operating conditions

– The possibility of unacceptable results with valid input, and

– The acceptability of effects when the information system is given invalid input

44



Performance constraints must be met– Are average response times met?

» (Hard real-time constraints rarely apply to information systems)

45



An information system is correct if it satisfies its specifications

Every information system has to be correct

But in addition, it must pass execution-based testing of – Utility– Reliability– Robustness, and– Performance

46


Who Should Perform Execution-Based Testing?

If a test case executes correctly, – Nothing is learned

If there is a failure– There is no doubt there is a fault

The aim of testing is come up with test cases that will highlight faults

Testing is therefore a destructive process

47



Programming is a creative process

Asking programmer to test a module he or she has implemented means– Asking him or her to execute the module in such a way

that a failure (incorrect behavior) ensues

This goes against the creative instincts of programmers

48



Programmers should not test their own modules– Testing that module requires the creator to perform a

destructive act and attempt to destroy that creation– Also, the programmer may have misunderstood some

aspect of the design or specification document» If testing is done by someone else, such faults may be discovered

49



The programmer desk checks the design before coding it

Then, he or she executes the module using test data– Probably the same test data that were used to desk check

the design

Next, the programmer tests the robustness of the module by running incorrect data

50



When the programmer is satisfied that the module is operating correctly– Systematic execution-based testing commences

This systematic testing should not be performed by the programmer

Independent testing must be performed by the quality assurance group– Quality assurance professionals must report to their own

managers and thus protect their independence

51



Performing systematic execution-based testing– An essential part of a test case is a statement of the

expected output before the test is executed» Both the test data and the expected results of that test must be

recorded

– After the test has been performed, the actual results should be recorded and compared with the expected results

– Recording must be done in machine-readable form» For later regression testing during maintenance

52


When Testing Stops

After many years of maintenance, an information system may lose its usefulness– It is decommissioned and removed from service

Only at that point, when the information system has been irrevocably discarded, can testing stop

slide 13.1 copyright © 2004 by the mcgraw-hill companies, inc. all rights reserved. an introduction...

Documents

testing slide

term testing

methodical testing

continual testing

testing stops

mcgrawhill companies

term test workflow slide

separate testing phase