Internet Software Internet Software DevelopmentDevelopment

Testing and InspectionsTesting and Inspections

Paul J KrausePaul J Krause

Software: benefits and risksSoftware: benefits and risks

Instruction 0x30da4b referenced memory at 0x04de

The Drive for Software QualityThe Drive for Software Quality

Low Defect Rates

High UserSatisfaction

But Testing is:-But Testing is:-

Time consumingTime consuming writing test caseswriting test cases executing test casesexecuting test cases

Error proneError prone

But Testing is:-But Testing is:-

Time consumingTime consuming writing test caseswriting test cases executing test casesexecuting test cases

Error proneError prone

Hazardous quality definitionHazardous quality definition

Quality means conformance to Quality means conformance to requirementsrequirements

Requirements contain > 15% of software errors.Requirements contain > 15% of software errors. Requirements grow at > 2% per month.Requirements grow at > 2% per month.

Do you conform to requirements errors?Do you conform to requirements errors? Do you conform to totally new requirements?Do you conform to totally new requirements? Whose requirements are you trying to satisfy?Whose requirements are you trying to satisfy?

The World is InformalThe World is Informal

“… “… one of the greatest difficulties in software one of the greatest difficulties in software development is formalisationdevelopment is formalisation

““- capturing in symbolic representation a worldly - capturing in symbolic representation a worldly computational problem so that statements computational problem so that statements obtained by following symbolic manipulation are obtained by following symbolic manipulation are useful statements once translated back into the useful statements once translated back into the real world.real world.

The formalisation problem is the essence of The formalisation problem is the essence of requirements engineering.”requirements engineering.”

W L ScherlisW L Scherlis

Errors, defects, failuresErrors, defects, failures

ErrorError DefectDefect FailureFailureIntroduces a May result in one or more

Effective and Efficient TestingEffective and Efficient Testing

To test To test effectivelyeffectively, you must use a strategy that , you must use a strategy that uncovers as many defects as possible. uncovers as many defects as possible.

To test To test efficientlyefficiently, you must find the largest , you must find the largest possible number of defects using the fewest possible number of defects using the fewest possible testspossible tests Testing is like detective workTesting is like detective work::

• The tester must try to understand how programmers and The tester must try to understand how programmers and designers think, so as to better find defectsdesigners think, so as to better find defects..

• The tester must not leave anything uncovered, and must be The tester must not leave anything uncovered, and must be suspicious of everythingsuspicious of everything..

• It does not pay to take an excessive amount of time; tester It does not pay to take an excessive amount of time; tester has to be has to be efficientefficient..

Black-box testingBlack-box testing

Testers provide the system with inputs and Testers provide the system with inputs and observe the outputsobserve the outputs They can see none of: They can see none of:

• The source codeThe source code• The internal dataThe internal data• Any of the design documentation describing the Any of the design documentation describing the

system’s internalssystem’s internals But they But they cancan see: see:

• Requirements documents and/or specificationsRequirements documents and/or specifications

Glass-box testingGlass-box testing

Also called ‘white-box’ or ‘structural’ testingAlso called ‘white-box’ or ‘structural’ testing

Testers have access to the system design Testers have access to the system design They can They can

• Examine the design documents Examine the design documents • View the codeView the code• Observe at run time the steps taken by algorithms and their Observe at run time the steps taken by algorithms and their

internal datainternal data Individual programmers often informally employ glass-Individual programmers often informally employ glass-

box testing to verify their own codebox testing to verify their own code

But they are testing what is implemented,and not necessarily what is required!

Recommended PracticeRecommended Practice

Use Black Box techniques for initial test Use Black Box techniques for initial test designdesign Check all requirements are coveredCheck all requirements are covered

Analyse the code or design to see to what Analyse the code or design to see to what extent the logical structure of the software extent the logical structure of the software under test is coveredunder test is covered

Intelligently add in test cases to increase Intelligently add in test cases to increase the structural coverage up to some agreed the structural coverage up to some agreed level level

Test design techniquesTest design techniques

Equivalence partitioningEquivalence partitioning ReducesReduces the number of test cases by the number of test cases by

selecting “representative” input valuesselecting “representative” input values Boundary value analysisBoundary value analysis

IncreasesIncreases effectiveness of test case design by effectiveness of test case design by focusing on “high risk” input valuesfocusing on “high risk” input values

These are applicable to Black Box testing These are applicable to Black Box testing as they base test design on the input as they base test design on the input values that are values that are requiredrequired to be handled to be handled

Equivalence classesEquivalence classes It is inappropriate to test by It is inappropriate to test by brute forcebrute force, using , using every every

possiblepossible input value input value Takes a huge amount of time Takes a huge amount of time Is impractical Is impractical Is pointless!Is pointless!

Divide the possible inputs into groups which you Divide the possible inputs into groups which you believe will be treated similarly by all algorithms. believe will be treated similarly by all algorithms. Such groups are called Such groups are called equivalence classesequivalence classes. . A tester needs only to run one test per equivalence classA tester needs only to run one test per equivalence class The tester has to The tester has to

• understand the required input, understand the required input, • appreciate how the software may have been designedappreciate how the software may have been designed

Examples of equivalence Examples of equivalence classesclasses

Valid input is a month number (1-12)Valid input is a month number (1-12) Equivalence classes are: [-∞..0], [1..12], [13.. Equivalence classes are: [-∞..0], [1..12], [13..

∞]∞]

Valid input is one of ten strings Valid input is one of ten strings representing a type of fuelrepresenting a type of fuel Equivalence classes areEquivalence classes are

• 10 classes, one for each string10 classes, one for each string• A class representing all other strings (Carefull!)A class representing all other strings (Carefull!)

Testing at boundaries of Testing at boundaries of equivalence classesequivalence classes

More errors in software occur at the boundaries More errors in software occur at the boundaries of equivalence classes of equivalence classes

The idea of equivalence class testing should be The idea of equivalence class testing should be expanded to specifically test values at the expanded to specifically test values at the extremes of each equivalence classextremes of each equivalence class E.g. The number 0 often causes problems E.g. The number 0 often causes problems

E.g.E.g.: If the valid input is a month number (1-12): If the valid input is a month number (1-12) Test equivalence classes as beforeTest equivalence classes as before Test 0, 1, 12 and 13 as well as very large positive and Test 0, 1, 12 and 13 as well as very large positive and

negative valuesnegative values

Detecting specific categories Detecting specific categories of defectsof defects

A tester must try to uncover any defects A tester must try to uncover any defects the other software engineers might have the other software engineers might have introduced. introduced. This means designing tests that explicitly try This means designing tests that explicitly try

to catch a range of specific types of defects to catch a range of specific types of defects that commonly occurthat commonly occur

Defects in Ordinary Defects in Ordinary AlgorithmsAlgorithms

Incorrect logical conditionsIncorrect logical conditions Performing a calculation in the wrong part of a Performing a calculation in the wrong part of a

control constructcontrol construct Not terminating a loop or recursionNot terminating a loop or recursion Not setting up the correct preconditions for an Not setting up the correct preconditions for an

algorithmalgorithm Not handling null conditionsNot handling null conditions Off-by-one errorsOff-by-one errors … …

Defects in Timing and Defects in Timing and Co-ordinationCo-ordination

Deadlock and livelockDeadlock and livelock DefectsDefects: :

• A deadlock is a situation where two or more A deadlock is a situation where two or more threads are stopped, waiting for each other to do threads are stopped, waiting for each other to do something.something.

The system is hung The system is hung

• Livelock is similar, but now the system can do Livelock is similar, but now the system can do some computations, but can never get out of some some computations, but can never get out of some states.states.

Defects in Timing and Co-Defects in Timing and Co-ordinationordination

Deadlock and livelockDeadlock and livelock Testing strategiesTesting strategies: :

• Deadlocks and livelocks occur due to unusual Deadlocks and livelocks occur due to unusual combinations of conditions that are hard to combinations of conditions that are hard to anticipate or reproduce. anticipate or reproduce.

• It is often most effective to use It is often most effective to use inspectioninspection to detect to detect such defects, rather than testing alone.such defects, rather than testing alone.

• However, when testing:However, when testing: Vary the time consumption of different threadsVary the time consumption of different threads.. Run a large number of threads concurrently.Run a large number of threads concurrently. Deliberately deny resources to one or more threadsDeliberately deny resources to one or more threads..

Example of deadlockExample of deadlock

B:ThreadA:Thread

waitingto lock O:

waitingto lock P:

lock

P:O:

lock

Defects in Timing and Co-Defects in Timing and Co-ordinationordination

Critical racesCritical races DefectsDefects::

• One thread experiences a failure because another One thread experiences a failure because another thread interferes with the ‘normal’ sequence of thread interferes with the ‘normal’ sequence of events.events.

Testing strategiesTesting strategies:: • It is particularly hard to test for critical races using It is particularly hard to test for critical races using

black box testing alone.black box testing alone.• One possible, although invasive, strategy is to One possible, although invasive, strategy is to

deliberately slow down one of the threadsdeliberately slow down one of the threads..• Use inspection.Use inspection.

Example of critical raceExample of critical race

A:Thread Data: B:Thread

get

set

a) Normal

A:Thread Data: B:Thread

get

set

b) Abnormal due to delay in thread A

Semaphore and Semaphore and synchronizationsynchronization

Critical races can be prevented by Critical races can be prevented by lockinglocking data so that they cannot be accessed by data so that they cannot be accessed by other threads when they are not readyother threads when they are not ready One widely used locking mechanism is called One widely used locking mechanism is called

a a semaphoresemaphore. . In Java, the In Java, the synchronizedsynchronized keyword can be keyword can be

used.used.• It ensures that no other thread can access an It ensures that no other thread can access an

object until the synchronized method terminatesobject until the synchronized method terminates..

Example of a synchronized Example of a synchronized methodmethod

A:Thread Data: B:Thread

get

get

a) Abnormal: The value put by thread A is immediately overwritten by the value put by thread B.

putcalc

putcalc

A:Thread Data: B:Thread

get

get

putcalc

putcalc

waiting for A: to complete its synchronized operation

b) The problem has been solved by accessing the data using synchronized methods

InspectionsInspections

An inspection is an activity in which one or An inspection is an activity in which one or more people systematicallymore people systematically Examine source code or documentation, Examine source code or documentation,

looking for defects. looking for defects. Normally, inspection involves a meeting...Normally, inspection involves a meeting...

• Although participants can also inspect alone at Although participants can also inspect alone at their desks.their desks.

Roles on inspection teamsRoles on inspection teams

The The authorauthor The The moderatormoderator. .

Calls and runs the meeting.Calls and runs the meeting. Makes sure that the general principles of inspection are adhered to. Makes sure that the general principles of inspection are adhered to.

The The secretarysecretary. . Responsible for recording the defects when they are found. Responsible for recording the defects when they are found. Must have a thorough knowledge of software engineeringMust have a thorough knowledge of software engineering..

ParaphrasersParaphrasers. . Step through the document explaining it in their own words.Step through the document explaining it in their own words.

Principles of inspectingPrinciples of inspecting

Inspect the most important documents of all typesInspect the most important documents of all types code, design documents, test plans and requirementscode, design documents, test plans and requirements

Choose an effective and efficient inspection teamChoose an effective and efficient inspection team between two and five peoplebetween two and five people Including experienced software engineers Including experienced software engineers

Require that participants prepare for inspectionsRequire that participants prepare for inspections They should study the documents prior to the meeting and come They should study the documents prior to the meeting and come

prepared with a list of defectsprepared with a list of defects

Only inspect documents that are readyOnly inspect documents that are ready Attempting to inspect a very poor document will result in defects Attempting to inspect a very poor document will result in defects

being missedbeing missed

Principles of inspectingPrinciples of inspecting

Avoid discussing how to fix defectsAvoid discussing how to fix defects Fixing defects can be left to the authorFixing defects can be left to the author

Avoid discussing style issuesAvoid discussing style issues Issues like are important, but should be Issues like are important, but should be

discussed separatelydiscussed separately Do not rush the inspection processDo not rush the inspection process

A good speed to inspect isA good speed to inspect is• 200 lines of code per hour (including comments)200 lines of code per hour (including comments)• or ten pages of text per houror ten pages of text per hour

Principles of inspectingPrinciples of inspecting

Avoid making participants tiredAvoid making participants tired It is best not to inspect for more than two hours at a It is best not to inspect for more than two hours at a

time, or for more than four hours a daytime, or for more than four hours a day

Keep and use logs of inspectionsKeep and use logs of inspections You can also use the logs to track the quality of the You can also use the logs to track the quality of the

design process design process

Re-inspect when changes are madeRe-inspect when changes are made You should re-inspect any document or code that is You should re-inspect any document or code that is

changed more than 20%changed more than 20%