software testing overviewsoftware testing overview · software testing overviewsoftware testing...

58
Software Testing Overview Software Testing Overview Dr. Andrea Arcuri Si l R hLb Simula Research Laboratory Oslo, Norway [email protected] Based on slides provided by Prof. Lionel Briand Lionel Briand 2009 1

Upload: others

Post on 25-Jun-2020

23 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Software Testing OverviewSoftware Testing Overview

Dr. Andrea ArcuriSi l R h L bSimula Research Laboratory

Oslo, Norway, [email protected]

Based on slides provided by Prof. Lionel Briand

Lionel Briand 20091

Page 2: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Software Testing Overview:Software Testing Overview:Part I

Lionel Briand 20092

Page 3: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Software has become prevalent in all aspects of our livesSoftware has become prevalent in all aspects of our lives

Lionel Briand 20093

Page 4: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Qualities of Software Products

• Correctness • Repairability• Reliability• Robustness

• Evolvability• Reusability

• Performance• User Friendliness

y• Portability• Understandability• User Friendliness

• Verifiability• Understandability• Interoperability

• Maintainability

Lionel Briand 20094

Page 5: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Pervasive Problems• Software is commonly delivered late, way over

budget, and of unsatisfactory qualityg , y q y• Software validation and verification are rarely

systematic and are usually not based on sound, well-defined techniques

• Software development processes are commonly bl d ll dunstable and uncontrolled

• Software quality is poorly measured, monitored, d t ll dand controlled.

• Software failure examples: http://www cs bc edu/~gtan/bug/softwarebug html

Lionel Briand 20095

http://www.cs.bc.edu/~gtan/bug/softwarebug.html

Page 6: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Examples of Software Failures• Communications: Loss or corruption of

communication media, non delivery of data.• Space Applications: Lost lives launch delays e gSpace Applications: Lost lives, launch delays, e.g.,

European Ariane 5 shuttle, 1996: – From the official disaster report: “Due to a

malfunction in the control software, the rocket veered off its flight path 37 seconds after launch.”

• Defense and Warfare: Misidentification of friend or foe.

• Transportation: Deaths, delays, sudden acceleration, inability to brake.

• Electric Power: Death injuries power outages

Lionel Briand 20096

Electric Power: Death, injuries, power outages, long-term health hazards (radiation).

Page 7: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Examples of Software FailuresExamples of Software FailuresExamples of Software Failures Examples of Software Failures (cont.)( )

• Money Management: Fraud, violation of privacy, shutdown of stock exchanges and banks negative interest ratesstock exchanges and banks, negative interest rates.

• Control of Elections: Wrong results (intentional or non-intentional).

• Control of Jails: Technology-aided escape attempts and successes, failures in software-controlled locks.

• Law Enforcement: False arrests and imprisonmentsLaw Enforcement: False arrests and imprisonments.

Lionel Briand 20097

Page 8: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Ariane 5 – ESA

On June 4, 1996, the flight of the Ariane 5 launcher ended in a failurefailure.

Only about 40 seconds afterOnly about 40 seconds afterinitiation of the flightsequence, at an altitude ofabout 3,700 m, the launcherveered off its flight path,broke up and exploded

Lionel Briand 20098

broke up and exploded.

Page 9: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Ariane 5 Root CauseAriane 5 – Root Cause• Source: ARIANE 5 Flight 501 Failure, Report by the Inquiry g , p y q y

BoardA program segment for converting a floating point number to a i d 16 bit i t t d ith i t d t l t idsigned 16 bit integer was executed with an input data value outside

the range representable by a signed 16-bit integer. This run time error (out of range overflow) which arose in bothThis run time error (out of range, overflow), which arose in both the active and the backup computers at about the same time, was detected and both computers shut themselves down. This resulted in the total loss of attitude control. The Ariane 5 turned uncontrollably and aerodynamic forces broke the vehicle apartapart. This breakup was detected by an on-board monitor which ignited the explosive charges to destroy the vehicle in the air. Ironically,

Lionel Briand 20099

the result of this format conversion was no longer needed after lift off.

Page 10: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Ariane 5 – Lessons Learned• Adequate exception handling and redundancy strategies

(real function of a backup system, degraded modes?)• Clear, complete, documented specifications (e.g.,

preconditions, post-conditions)• But perhaps more importantly: usage-based testing

(based on operational profiles), in this case actual Ariane 5 trajectories

• Note this was not a complex, computing problem, but a deficiency of the software engineering practices in place …

Lionel Briand 200910

Page 11: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

F 18 hF-18 crash• An F-18 crashed because of a missing exception

condition: An if ... then ... block without the else clause that was thought could not possibly arise.

• In simulation, an F-16 program bug caused the virtual plane to flip over whenever it crossed the equator, as a result of a missing minus sign to indicate south latitude.

Lionel Briand 200911

Page 12: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Fatal Therac-25 Radiation

• In 1986, a man in Texas received between 16,500-25 000 radiations in less than 10 sec over an area25,000 radiations in less than 10 sec, over an area of about 1 cm.

• He lost his left arm and died of complications 5• He lost his left arm, and died of complications 5 months later.

Lionel Briand 200912

Page 13: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Power Shutdown in 2003

508 generating units

Power Shutdown in 2003

508 generating units and 256 power

plants shut down

Affected 10 million people in Ontario,

Canada

Affected 40 million people in 8 US

statesstates

Financial losses of$6 Billion USD$

The alarm system in the energy management system failed due to a software error and operators were not informed of the power

Lionel Briand 200913

to a software error and operators were not informed of the power overload in the system

Page 14: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Consequences of Poor Quality• Standish Group surveyed 350 companies, over 8000

projects in 1994projects, in 1994• 31% cancelled before completed, 9-16% were delivered

within cost and budget• US study (1995): 81 billion US$ spend per year for failing

software development projectsNIST d (2002) b $ 59 5 billi E li• NIST study (2002): bugs cost $ 59.5 billion a year. Earlier detection could save $22 billion.

Lionel Briand 200914

Page 15: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Quality AssuranceQuality Assurance• Uncover faults in the documents where they areUncover faults in the documents where they are

introduced, in a systematic way, in order to avoid ripple effects. Systematic, structured reviews of software d t f d t i tidocuments are referred to as inspections.

• Derive, in a systematic way, effective test cases to uncover faultsfaults

• Automate testing and inspection activities, to the maximum extent possible

• Monitor and control quality, e.g., reliability, maintainability, safety, across all project phases and activitiesactivities

• All this implies the quality measurement of SW products and processes

Lionel Briand 200915

p

Page 16: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Dealing with SW FaultsDealing with SW FaultsF lt H dliFault Handling

F lt T lFault Avoidance Fault ToleranceFault Detection

Configuration

AtomicTransactions

ModularRedundancyInspectionsDesign

Methodology

Testing Debugging

Verification ConfigurationManagement

Component Integrationi

Systemi

Correctnessi

Performancei

Lionel Briand 200916

Testing Testing Testing Debugging Debugging

Page 17: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Testing Definition

• SW Testing: Techniques to execute programs with the intent of finding as many defects as possible and/or gaining sufficient confidence in the software system under test.– “Program testing can show the presence ofProgram testing can show the presence of

bugs, never their absence” (Dijkstra)

Lionel Briand 200917

Page 18: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Basic Testing Definition• Errors: People commit errors• Fault: A fault is the result of an error in the software

doc mentation code etcdocumentation, code, etc.• Failure: A failure occurs when a fault executes• Many people use the above three terms inter changeably It• Many people use the above three terms inter-changeably. It

should be avoided• Incident: Consequences of failures – Failure occurrence q

may or may not be apparent to the user• The fundamental chain of SW dependability threats:

E rror Fault Failurepropagation c aus ation

. . .Inc identres ults in

Lionel Briand 200918

Page 19: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Why is SW testing important?• According to some estimates: ~50% of development

costs• A study by (the American) NIST in 2002:

The annual national cost of inadequate testing is as– The annual national cost of inadequate testing is as much as $59 Billion US!Th i i l d “Th E i I f– The report is titled: “The Economic Impacts of Inadequate Infrastructure for Software Testing”

Lionel Briand 200919

Page 20: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

TestingTestingDefinitions & Objectivesj

Lionel Briand 200920

Page 21: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Test Stubs and Drivers• Test Stub: Partial implementation of a component on which a unit under test

depends. T t S t b

C om ponent a C om ponent b

U nder Tes t

Tes t S tub

D epends

• Test Driver: Partial implementation of a component that depends on a unit under

U nder Tes t

Test Driver: Partial implementation of a component that depends on a unit under test.

C om ponent j C om ponent k

Tes t D riv er

D epends

• Test stubs and drivers enable components to be isolated from the rest of the

C om ponent j C om ponent k

U nder Tes t

Lionel Briand 200921

• Test stubs and drivers enable components to be isolated from the rest of the system for testing.

Page 22: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Summary of DefinitionsTest suite

exercises is revised by

Test case CorrectionComponent

exercises s e sed by

* * 1…n

Test stubfinds

repairs

* *

*

*

Test driver

repairs

*

*

is caused by

* *Failure Error

is caused by

***

Fault

Lionel Briand 200922

Page 23: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Motivations• No matter how rigorous

we are, software is going to be faulty Limited resourcesy

• Testing represent a substantial percentage of software development Time

Money Peoplexpertise

costs and time to market• Impossible to test under

all operating conditions –b d i l t

Money e

based on incomplete testing, we must gain confidence that the system has the desired behaviors e des ed be v o

• Testing large systems is complex – it requires strategy and technology-

Lionel Briand 200923

and is often done inefficiently in practice

Page 24: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

The Testing Dilemma

All Software System

Available testing

resources

Potentially functionality

ythousands of itemsof items to test

Lionel Briand 200924Faulty functionality

Page 25: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

T ti P O iTesting Process OverviewSW Representation(e.g., models, requirements)

D i T t

SW Code

Derive Test cases

Execute Test cases

Estimate Expected

Execute Test casesResults

Get Test Results

CompareTest Oracle[Test Result==Oracle]

[Test Result!=Oracle]

Lionel Briand 200925

Page 26: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Qualities of Testing

• Effective at uncovering faults• Help locate faults for debugging• Repeatable so that a precise understandingRepeatable so that a precise understanding

of the fault can be gained• Automated so as to lower the cost and• Automated so as to lower the cost and

timescaleS i b di bl i• Systematic so as to be predictable in terms of its effect on dependability

Lionel Briand 200926

Page 27: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Continuity Property• Problem: Test a bridge ability to sustain a

certain weight• Continuity Property: If a bridge can sustain aContinuity Property: If a bridge can sustain a

weight equal to W1, then it will sustain any weight W2 <= W1E ti ll ti it t ll• Essentially, continuity property= small differences in operating conditions should not result in dramatically different behavior

• BUT, the same testing property cannot be applied when testing software, h ?why?

• In software, small differences in operating conditions can result in dramatically different behavior (e.g., value boundaries)

Lionel Briand 200927

y ( g , )• Thus, the continuity property is not applicable to software

Page 28: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Subtleties of SoftwareSubtleties of Software DependabilityDependability

• Dependability: Correctness, reliability, safety, robustness

• A program is correct if it obeys its specification.• Reliability is a way of statistically approximating

correctness.• Safety implies that the software must always

display a safe behavior, under any condition.• A system is robust if it acts reasonably in severe,

unusual or illegal conditions.

Lionel Briand 200928

Page 29: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Subtleties of SoftwareSubtleties of Software Dependability IIDependability II

• Correct but not safe or robust: the specification is i dinadequate

• Reliable but not correct: failures rarely happen • Safe but not correct: annoying failures may

happenpp• Reliable and robust but not safe: catastrophic

failures are possiblefailures are possible

Lionel Briand 200929

Page 30: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Software DependabilitySoftware Dependability Ex: Traffic Light ControllerEx: Traffic Light Controller

• Correctness, Reliability:The system should let traffic pass according to the correct pattern and centralThe system should let traffic pass according to the correct pattern and central scheduling on a continuous basis.• Robustness:The system should provide degraded functionality in the presence ofThe system should provide degraded functionality in the presence of abnormalities.• Safety:It should never signal conflicting greensIt should never signal conflicting greens.

An example degraded function: the line to central controlling is cut-off and a p g gdefault pattern is then used by local controller.

Lionel Briand 200930

Page 31: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Dependability Needs Vary• Safety-critical applications

flight control systems have strict safety requirements– flight control systems have strict safety requirements– telecommunication systems have strict robustness

requirementsq• Mass-market products

– dependability is less important than time to market p y p• Can vary within the same class of products:

– reliability and robustness are key issues for multi-user operating systems (e.g., UNIX) less important for single users operating systems (e.g., Windows or MacOS)

Lionel Briand 200931

MacOS)

Page 32: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Fundamental Principles

Lionel Briand 200932

Page 33: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Exhaustive TestingExhaustive TestingE h ti t ti i t ti ft t i ll• Exhaustive testing, i.e., testing a software system using all the possible inputs, is most of the time impossible.

• Examples:• Examples:– A program that computes the factorial function (n!=n.(n-1).(n-2)…1)

• Exhaustive testing = running the program with 0, 1, 2, …, 100, g g p g , , , , ,… as an input!

– A compiler (e.g., javac)E h i i i h (J ) il i h• Exhaustive testing = running the (Java) compiler with any possible (Java) program (i.e., source code)

Lionel Briand 200933

Page 34: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Input Equivalence Classes

General principle to reduce the number of inputs Testing criteria group input elements into (equivalence)

classesO i i l d i h l ( i f– One input in selected in each class (notion of test coverage)

Input Domain

tc4tc5

tc1 tc3tc6

Lionel Briand 200934

tc2

Page 35: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

T t CTest CoverageS ft R t tiSoftware Representation

(Model) Associated CriteriaTest cases must coverTest cases must cover all the … in the model

Test Data

Representation of • the specification Black Box Testing• the specification Black-Box Testing

• the implementation White-Box Testing

Lionel Briand 200935

Page 36: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Complete Coverage: White-Boxif x > y then

Max := x;;else

Max :=x ; // fault!d ifend if;

{x=3, y=2; x=2, y=3} can detect the error, more “coverage”{ , y ; , y } , g{x=3, y=2; x=4, y=3; x=5, y=1} is larger but cannot detect it

Testing criteria group input domain elements into (equivalence)• Testing criteria group input domain elements into (equivalence) classes (control flow paths here)

• Complete coverage attempts to run test cases from each class

Lionel Briand 200936

Page 37: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Control Flow Coverage (CFG) Control Flow Coverage (CFG) -- ExampleExample

Greatest common divisor (GCD) program

read(x);read(y); read(y);

while x y loopif x>y then x = y

x y

x := x – y;else

x<=y x > y

y := y – x;end if;

end loop;end loop;gcd := x;

Lionel Briand 200937

Page 38: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Control Flow Coverage (CFG) Control Flow Coverage (CFG) -- DefinitionsDefinitions

• Directed graph• Nodes are blocks of sequential q

statements• Edges are transfers of control x<=y x > y

x = y

x y

g• Edges may be labeled with

predicate representing the

x< y x > y

p p gcondition of control transfer

• There are several conventions for flow graph models with subtle differences (e.g., hierarchical CFG CFG )

Lionel Briand 200938

CFGs, concurrent CFGs)

Page 39: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Basics of CFG: BlocksBasics of CFG: Blocks

If-Then-Else While loop SwitchIf-Then-Else While loop Switch

Lionel Briand 200939

Page 40: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Testing Coverage of Control flow

• As a testing strategy, we may want to ensure that testing exercises control flow:control flow:– Statement/Node Coverage– Edge/Branch CoverageEdge/Branch Coverage– Condition Coverage

Path Coverage < >x = y

x y

– Path Coverage x<=y x > y

Lionel Briand 200940

Page 41: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Complete Coverage: Black-Box• Specification of Compute Factorial Number: If the input value n is < 0, then an

appropriate error message must be printed. If 0 <= n < 20, then the exact value of n! must be printed If 20 <= n < 200 then an approximate value of n! must be printed inmust be printed. If 20 <= n < 200, then an approximate value of n! must be printed in floating point format, e.g., using some approximate method of numerical calculus. The admissible error is 0.1% of the exact value. Finally, if n>=200, the input can be rejected by printing an appropriate error message.

• Because of expected variations in behavior, it is quite natural to divide the input domain into the classes {n<0}, {0<= n <20}, {20 <= n < 200} { 200} W t t f h l200}, {n >= 200}. We can use one or more test cases from each class in each test set. Correct results from one such test set support the assertion that the program will behave correctly for any other class

al e b t there is no g arantee!value, but there is no guarantee!

Lionel Briand 200941

Page 42: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Black vs. White Box TestingSystem

Specification

I l t tiImplementation

Missing functionality: Cannot be revealed by white-box t h i

Unexpected functionality: Cannot be revealed by black-box t h i

Lionel Briand 200942

techniques techniques

Page 43: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

White-box vs. Black-box Testing

• Black box+ Check conformance ith

•White box+ It allows you to be+ Check conformance with

specifications+ It scales up (different

t h i t diff t

+ It allows you to be confident about code coverage of testing+ It i b d t ltechniques at different

granularity levels)– It depends on the

ifi i i d

+ It is based on control or data flow code analysis– It does not scale up ( l li bl ispecification notation and

degree of detail– Do not know how much of

(mostly applicable at unit and integration testing levels)

the system is being tested– What if the software

performed some

– Unlike black-box techniques, it cannot reveal missing functionalities (part

f h ifi i h i

Lionel Briand 200943

punspecified, undesirable task?

of the specification that is not implemented)

Page 44: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Software Testing Overview:Software Testing Overview:Part II

Lionel Briand 200944

Page 45: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Practical Aspects

Lionel Briand 200945

Page 46: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Many Causes of Failures

• The specification may be wrong or have a missing requirement

• The specification may contain aThe specification may contain a requirement that is impossible to implement given the prescribed software and hardwaregiven the prescribed software and hardware

• The system design may contain a fault• The program code may be wrong

Lionel Briand 200946

Page 47: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Test Organization

• May different potential causes of failure, Large i i l lsystems -> testing involves several stages

• Module, component, or unit testing• Integration testing• Function testFunction test• Performance test

A• Acceptance test• Installation test

Lionel Briand 200947

Page 48: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Unitode

Design System Other UserUnittest

onen

t co Design

descriptionsSystem

functionalspecifications

Othersoftware

specifications

Customerrequirements

Userenvironment

Unitt t

Com

pode test

Integration Function Performance Acceptance Installationonen

t co

Integrationtest

Functiontest

Performancetest

Acceptancetest

Installationtest

Com

po ..

code . Integrated

modulesFunctioning

systemVerified,validatedsoftware

Acceptedsystem

Unittest

pone

nt

SYSTEMIN USE!Pfleeger, 1998

Lionel Briand 200948C

om

IN USE!g ,

Page 49: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Unit Testing• (Usually) performed by each developer.• Scope: Ensure that each module (i.e., class, subprogram) has been

implemented correctly. • Often based on White-box testing.

• A unit is the smallest testable part of an applicationTest

• A unit is the smallest testable part of an application. • In procedural programming, a unit may be an individual

subprogram, function, procedure, etc. p g , , p ,• In object-oriented programming, the smallest unit is a method;

which may belong to a base/super class, abstract class or d i d/ hild l

Lionel Briand 200949

derived/child class.

Page 50: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Integration/Interface Testing• Performed by a small team.• Scope: Ensure that the interfaces between components (which• Scope: Ensure that the interfaces between components (which

individual developers could not test) have been implemented correctly, e.g., consistency of parameters, file format

Test

• Test cases have to be planned, documented, and reviewed.P f d i l ti l ll ti f

Lionel Briand 200950

• Performed in a relatively small time-frame

Page 51: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

I t ti T ti F ilIntegration Testing FailuresI i f ll d l dIntegration of well tested components may lead to

failure due to:B d f h i f (b d i f• Bad use of the interfaces (bad interface specifications / implementation)W h h i h b h i / f l d• Wrong hypothesis on the behavior/state of related modules (bad functional specification / implementation) e g wrong assumption aboutimplementation), e.g., wrong assumption about return value

• Use of poor drivers/stubs: a module may behave• Use of poor drivers/stubs: a module may behave correctly with (simple) drivers/stubs, but result in failures when integrated with actual (complex)

Lionel Briand 200951

failures when integrated with actual (complex) modules.

Page 52: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

System TestingSystem Testing• Performed by a separate group within the organization (Most of y p g p g (

the times).

• Scope: Pretend we are the end-users of the product. p p

• Focus is on functionality, but may also perform many other types of non-functional tests (e.g., recovery, performance).

Test

• Black-box form of testing, but code coverage can be monitored.

Lionel Briand 200952• Test case specification driven by system’s use-cases.

Page 53: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Differences among TestingDifferences among Testing ActivitiesActivities

Unit Testing Integration Testing System Testing

From modulespecifications

From interfacespecifications

From requirements specsspecifications

Visibilityof code details

specifications

Visibilityof integr. Struct.

p

No visibility of codeof code details

Complex scaffolding

of integr. Struct.

Somescaffolding

No drivers/stubs

Behavior of single modules

Interactions among modules

System functionalities

Lionel Briand 200953

g g

Pezze and Young, 1998

Page 54: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

System vs. Acceptance Testing• System testing

– The software is compared with the requirements p qspecifications (verification)

– Usually performed by the developers, who know the tsystem

• Acceptance testingTh ft i d ith th d– The software is compared with the end-user requirements (validation)

– Usually performed by the customer (buyer) who knowsUsually performed by the customer (buyer), who knows the environment where the system is to be used

– Sometime distinguished between - -testing for

Lionel Briand 200954

general purpose products

Page 55: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Testing through the Lifecycle• Much of the life-cycle development artifacts provides a

rich source of test datarich source of test data• Identifying test requirements and test cases early helps

shorten the development time• They may help reveal faults• It may also help identify early low testability specifications

d ior design

Analysis Design Implementation Testingy g p g

Lionel Briand 200955

Preparation Preparation for Testfor Test

Preparation Preparation for Testfor Test

Preparation Preparation for Testfor Test

TestingTesting

Page 56: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Life Cycle Mapping: V Model

Other name:Integrationtestingtesting

Other name:Unitt ti

Lionel Briand 200956

testing

Page 57: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

Testing Activities BEFORETesting Activities BEFORE CodingCoding

• Testing is a time consuming activityD i i d id if h• Devising a test strategy and identify the test requirements represent a substantial part of itPl i i i l• Planning is essential

• Testing activities undergo huge pressure as it is is d h d f h jrun towards the end of the project

• In order to shorten time-to-market and ensure a i l l f li l f QA l dcertain level of quality, a lot of QA-related

activities (including testing) must take place early in the development life cycle

Lionel Briand 200957

in the development life cycle

Page 58: Software Testing OverviewSoftware Testing Overview · Software Testing OverviewSoftware Testing Overview Dr. Andrea Arcuri Si l R h L bSimula Research Laboratory Oslo,,y Norway arcuri@simula.no

T ti t k ti itTesting takes creativityT i f i d di k ( h h l• Testing often viewed as dirty work (though less and less).T d l ff i h• To develop an effective test, one must have:

• Detailed understanding of the system • Knowledge of the testing techniques• Knowledge of the testing techniques• Skill to apply these techniques in an effective and efficient

manner

• Testing is done best by independent testers• Programmer often stick to the data set that makes

the program work • A program often does not work when tried by

Lionel Briand 200958

somebody else.