software testing part i - dis.uniroma1.itsantucci/sw_engineering/material/05_a_test_parti.pdf ·...

Software Testing part I

Lecturer: Giuseppe Santucci

Software testing

1. Basic concepts

2. Test that does not use the computer

3. Black box testing

4. White box testing

5. Test strategies

Test.2

1. – Basic concepts

• An example

• Test definition

• Testing principles

03Test.3

Test the following program

A Java program that:

• reads from the file "foo.txt", which is

generated by another program, three

integers, corresponding to the sides of a

triangle

• Print on the screen the triangle type:

– isosceles,

– scalene, or

– equilateral

03Test.4

Prepare some files to test it

// File Triangolo.java

import java.io.*;

public class Triangolo {

public static void main (String[] args) throws IOException {

String nome_file = “foo.txt";

FileInputStream istream = new FileInputStream(nome_file);

BufferedInputStream bis = new BufferedInputStream(istream);

int first = InOut.readInt(bis),

second = InOut.readInt(bis),

third = InOut.readInt(bis);

bis.close();

System.out.println(“Sides: "+first+' '+second+' '+third);

if (first == secondo && second == third)

System.out.println("Equilateral");

else if (first== second || second == third)

System.out.println("Isosceles");

else

System.out.println("Scalene");

}

}

Test.5

Did you considered the following cases?

1. A valid scalene (2,5,10 is wrong)

2. A valid equilateral

3. A valid isosceles (2,2,4 is wrong)

4. Three permutations of an isosceles (3,3,4 / 3,4,3 / 4,3,3)

5. One side equal to 0

6. A negative side

7. 3 integers >0 and one is the sum of the others two (1,2,3)

8. Test 7 permutations (1,2,3 / 1,3,2 / 3,1,2)

Test.6

Did you considered the following cases?(2)

9. 3 integers >0 and one is less than the sum of the

others two (2,5,10)

10.Test 9 permutations

11. 3 0s

12.One side as real

13.Only two sides

14.The file does not exist or it is not readable

15.Did you provide for each test the foreseen output?

Test.7

Results

According to Myers, [1979], on the basis of

the answers got from experienced

programmers, the media of Yes answers to

the previous questions is less than 8 (out of

15)

Testing, even a trivial program, is not a trivial

activity

Test.8

Moreover

• Myers does not consider the test in which the file

contains more than 3 integers

• or maxint ...

Test.9

• Revisions and other inspection activities are able to detect errors, but are not enough

• Detecting errors typically costs 30-40% of the total project (but the percentage may be much higher)

• testing is the process of using a program with the specific intent to find errors before delivery it to the end-user

Test.10

Testing definition

Test.11

Psychological issues

From the psychological point of view, it is a

destructive phase (all other phases, e.g., design,

are constructive):

1. the test is to run a program in order to discover

errors

2. a case testing is of good quality if it has a high

probability of discovering errors

3. a test case is successful if it found previously

unknown errors

03Test.12

General testing goals

Testing is the design and execution of test cases able

to detect errors:

Of different types

In a systematic way

As fast as possible

Using a predefined amount of resources

Testing has not the goal of demonstrating that the

software is error-free

Test.13

Testing limits

Testing cannot demonstrate that the software is

error-free

Example: Java program about triangles

One or more inputs: files with 3 integers (not

enough)

Number of different inputs: 32 bit integers

232232232=296

Time to execute all of them (brute force):

1 Million test/second (overestimated!)

276 seconds 251 years (about 1015)

Test.14

Testing principles

We have to define the output of a test case before

executing it

Ideally, we should get someone else to check our

work : another person is more likely to find the

errors

1. Psychological issues: finding error in own work

is a destructive process

2. Test cases will be driven by the knowledge of

the program structure

NOTE: it is not true for debugging

Test.15

Testing principles (2)

Test results have to be carefully analyzed

Test cases must be designed:

• For non valid and non foreseen

conditions

• For valid and foreseen conditions

Testing has to check if:

• The program does not what it is supposed

to do

• The program does what it should not to do

Test.16


Avoid “use and thrown away” use cases

• You lose your investment

• You have to reinvent the tests

• As a consequence, tesing a new release of the

sw is less rigorouse

Do not plan test cases assuming (hoping?) that

you will not find errors! You need to allocate time

and resources for debugging

Design tests in advance (e.g., when requirements

are ready, see XP)


Test.17

Number of found errors

Probability to find

other errors

The probability of the existence of errors in a

program grows with the number of errors already

found. The errors tend to cluster in certain sections

of code (Pareto principle: 80% of the errors were

due to 20% of code)

03Test.18


Test cases rise from and must be connected

with user requirements

We start from the single functions (unit test) till

encompass the overall program structure

Brute force test is not possible testing is a

creative and stimulating activity

Test.19


• Some empirical rules:

• A good test case is not redundant compared

to the previous ones (you risk to waste time

and resources)

• It is important to determine categories of test

cases

• A good test case is neither too easy nor too

complex

03Test.20

Design the SW with testing phase in mind

Testability: the sw is easy to test. It depends on

Observeability: test results are observable (e.g., internal

states and variables, the source code is available, …)

Controllability: testing can be automated. It is possible to fill

variables from outside and inspect them

Decomposability: level of modularity : it is possible to

independently test modules

Simplicity: only mandatory functions are implemented (not

gold plating), agreed usage of standard coding, modular

architecture

Pesticide paradox

• If the same tests are repeated over and over

again, eventually the same set of test cases will

no longer find any new bugs

• To overcome this pesticide paradox

– the test cases need to be regularly reviewed and

revised

– new and different tests need to be written to exercise

different parts of the software or system to potentially

find more defects

03Test.21

Overall testing strategies

Three different objectives:

• Highlight a pre-fixed number of defects, minimizing the cost

• To make the best use of a fixed budget

• Looking for the global minimum of the function of cost

03Test.22

2. – Testing without a computer

• Inspection and walkthrough

• Common errors

03Test.23

“Human based testing”

• Static code analysis• Such techniques are not substituting computer

based techniques and have to be used when the code is not running

• Experience shows that such techniques are very effective: – It seems that psychological aspects are not affecting

such a process– The sooner you discover an error the less the cost

03Test.24

03Test.25

Inspection e walkthrough

• the sw is examined by a group of people (3-5), including

the author(s)

• involved people have (typically) viewed the product before

the meeting

• the programmer comments, line by line, the program

• significant sets of input are selected

• The sw is simulated on each set

• the aim must be to find mistakes, not correct them (testing,

not debugging)

3. Black box testing

03Test.26

• Introduction

• Equivalence classes coverage

• Boundary value analysis (BVA)

03Test.27

Introduction

• Functional test (vs structural test)

• Uses the definition of the input-output behavior of programsto design test cases

• General principle: the design of test cases occurs

disregarding the internal structure of the sw and reasoning

only on requirements (test cases (should) must be designed

after requirement specification)

03Test.28

Introduction (2)

• (Typically, but not always) performed after thewhite box testing (not alternative butcomplementary to it)

• if properly conducted, can discover manyfunctional failures

• may also highlight the failure of some non-functional requirement (e.g., inadequateperformance)

Equivalence classes

Equivalence classes

• It is a common sense approach to testing

• Most testers practice it informally even though

they may not realize it

• The idea behind the technique is to partition a

set of input values into equivalence classes

– The system should handle them equivalently

Equivalence partitioning

• The equivalence-partitioning technique requires that

we need to test only one input value from each

equivalence class

• We are assuming that all the input values in one

class will be treated in the same way by the software

• If one input value in a class works, we assume all of

the input conditions in that partition will work

• Conversely, if one of the input value in a class does

not work, then we assume that none of the input

conditions in that class will work

Example 1/5

• A savings account in a bank earns a different rate of

interest depending on the balance in the account

• In order to test the software that calculates the

interest due, we can identify the ranges of balance

values that earn the different rates of interest

– A balance in the range $0 up to $100 has a 3% interest

rate

– A balance over $100 and up to $1000 has a 5% interest

rate

– Balances of $1000 and over have a 7% interest rate

Example 2/5

• We would identify four equivalence classes

– Three valid equivalence classes and one invalid class

• Important:

– Not only we test what is in our specification

– We also think about things that haven't been

specified

– In this case we have thought of the situation where

the balance is less than zero

– We haven’t identified an invalid class on the right

Invalid class Valid (for 3% interest) Valid (for 5%) Valid (for 7%)

-? -- -$0.01 $0.00 -- $100.00 $100.01 -- $1000.00 $1000.01 -- ?

Example 3/5

• Do you identify additional classes?


-$0.01 -- $0.00 $0.01 -- $100.00 $100.01 -- $1000.00 $1000.01 -- ?

Example 4/5

• Do you identify additional class?

• What about non-numeric numbers /format

errors?

• Overflow?


-$0.01 -- $0.00 $0.01 -- $100.00 $100.01 -- $1000.00 $1000.01 -- ?

Example 5/5

• When we say a class is invalid, it does not mean that it represents a value that cannot be entered by a user or a value that the user is not supposed to enter

• It just means that it is not one of the expected inputs for this particular field

• The software should correctly handle values from the invalid class – E.g. By replying with an error message such as Balance must be

at least $0.00

• Note also that the invalid class may be invalid only in the context of crediting interest payments.– E.g. An account that is overdrawn will require some different

action.

Identifying equivalence classes

• Distinguish between valid and invalid classes

• Classes are identified by carefully analyzing

input conditions

• From a sentence in the specification derive one

or more valid or invalid classes

Identifying Equivalence classes

• List the classes

• Each class has an identification code

• Useful criteria for partitioning

– Intervals of values

– Number of values

– Sets of values

– Constraints

Intervals of values 1/2

• If an input condition specifies an interval of

values, identify

– One valid class

• Values within the interval

– Two invalid classes

• Values below the minimum boundary

• Values above the maximum boundary

Intervals of values 2/2

Condition Valid class Invalid classes

Customer code

between 1 and 999

1<= Customer code<= 999

1

Customer code< 1

Customer code> 999

2

3

Number of values 1/2

• If an input condition specifies the number of

values, identify

– One valid class

• For a number of values between the minimum and the

maximum specified

– Two invalid classes

• One for values lower than the minimum

• One for values greater than the maximum

Number of values 2/2

Condition Valid class Invalid classes

The number of product must not exceed 6

1<= Number of products<= 6

4

Number of products< 1

Number of products> 6

5

6

Where this 1 comes from?

Constraints 1/2

• If an input condition specifies something like

must be, identify

– One valid class

• For elements that satisfy the constraint

– One invalid class

• For elements that brake the condition

Constraints 2/2

Condition Valid class Invalid class

The type of the product must be “Manufactured”

Product type= “Manufactured”

7

Product type!= “Manufactured”

8

Set of values 1/2

• If an input condition specifies a set of values,

identify

– A number of valid classes equal to the set cardinality

• If the element of the set are treated in the same way by the

system they can be grouped in the same partition

– One invalid class

• For elements that are not in the set

Set of values 2/2

Condition Valid class Invalid class

The type of customer

can be either “old” or “new”

Customer type=“new”

9 Customer type not in {“old”, “new”}

11

Customer type=“old”

10

Test cases design

• Valid classes

– To design a number of test cases sufficient to cover

all valid classes

– Constraint: to maximize the number of classes

covered by each test case (to minimize the number of

test cases)

• Note: sometimes such criterion may be in

contrast with easy localization of error

47

Valid classes: example

48

Conditions Valid Invalid

Equivalence classes

1. Customer code between 1 and 999

2. Number of products must not exceed 6

3. Type of product must be “Manufactured”

4. Customer type can be either “old” or “new”

1 <= Customer code<= 999 (1)

1<= Number of products<= 6 (4)

Product type= “Manufactured” (7)

Customer type=“new” (9)

Customer type=“old” (10)

Customer code<1 (2)

Customer code>999 (3)

Number of products< 1 (5)

Number of products> 6 (6)

Product type= “Imported” (8)

Customer type != {“new”, “old”} (11)

Test case:

Customer code = 5

Number of products = 4

Type of product =“Manufactured”

Customer type = “new”

Covers classes (1) (4) (7) (9)

Valid classes: example

49










Customer code<1 (2)




Product type= “Imported” (8)


Test case:

Customer code = 5

Number of products = 4

Type of product =“Manufactured”

Customer type = “old”

Covers classes (10)


Equivalence classes

Test cases design cont.d

• Invalid classes

– To design a number of test cases sufficient to cover

all invalid classes

– Constraint: each test has to cover one and only one

invalid class

• In some cases that is not possible and the test is designed

minimizing the number of covered invalid classes

• Invalid classes have to be covered individually in

order to avoid errors that hide each other

50

Invalid classes: example

51










Customer code<1 (2)




Product type!= “Manufactured” (8)


Partition

covered(2) (3) (5) (6) (8) (11)

Customer code 0 1000 7 7 7 66

Number of

products 1 1 0 9 3 4

Type of products Manufactured Manufactured Manufactured Manufactured Imported Manufactured

Customer type New New New New New recent


Equivalence classes

Equivalent classes for the triangle

program

• Criterion: interval of values– Every side has a positive value -> the interval has only the

minimum boundary

• Valid class(VCi):– VC1: 0 < a <= 100 VC2: 0< b <= 100 VC3: 0 < c <= 100

• Invalid classes(ICi)– IC4: a <= 0 IC5: b <= 0 IC6: c <= 0

– IC7: a > 100 IC8: b > 100 IC9: c > 100

• Note: – 100 is an arbitrary value, we could use MAXINT

– It is important to always define a maximum boundary• E.g., by using values that can be represented

52

Equivalent partitions for the triangle program

cont.d

• Criterion: number of values

– The txt file must contain exactly three integers

– I.e., min number = max number

• Valid class

– VC10: exactly three integers

• Non valid classes

– IC11: more than three integers

– IC12: less than three integers

53

Equivalent partitions for the triangle program

cont.d

• Criterion: set of values

– There are three types of triangles

• Valid classes

– VC13: equilateral triangle

– VC14: isosceles triangle

– VC15: scalene triangle

• Non valid class

– IC16: it is not a triangle

54

Equivalent classes for the triangle

program cont.d

55

• Criterion: constraints

– The txt file contains integers

• VC17 : the file contains integers

• IC18 : the file does not contain integers

– The file exists

• VC19 : the file exists

• VC20 : the file does not exist

Test cases for the “triangle program”

56

Test cases Covered equivalence classes Output

1) 4, 4, 4 VC1, VC2, VC3, VC10, VC13, VC17,

VC19

Equilateral triangle

2) 2, 2, 1 VC14 Isosceles triangle

3) 3, 4, 5 VC15 Scalene triangle

4) -3, 4, 5 IC4 Err: a non-positive

5) 3, -4, 5 IC5 Err: b non-positive

6) 3, 4, -5 IC6 Err: c non-positive

7) 150, 4, 5 IC7 Err: a greater than max

8) 3, 150, 5 IC8 Err: b greater than max

9) 3, 4, 150 IC9 Err: c greater than max

10) 3, 4, 5, 2 IC11 Err: too many input data

11) 3, 4 IC12 Err: too few input data

12) 1, 1, 5 IC16 Err: non-valid triangle

13) 3.0, 4, 5 IC18 Err: a is not integer

14) No file IC20 Err: the file does not exist

Boundary value analysis

03Test.57

Boundary value analysis (BVA)

• It is based on testing at the boundaries between

classes

• Note that we have both valid boundaries (in the

valid equivalence classes) and invalid

boundaries (in the invalid equivalence classes)

• Practical experience showed that boundary

value analysis lead to beneficial results during

testing

Boundary value analysis (BVA) cont.d

• You have to consider values:

– Directly on the boundaries

– Immediately above them

– Immediately below them

• of equivalence classes of input and output


• It means that we are looking for values “close” to one

boundary (not a formal definition)

• BVA differs from equivalence classes partitioning in two

aspects

– You choose one or more values around each

boundary as representatives of the equivalence

partition

– Test cases are designed by considering also results

(output)

• Such technique requires a good knowledge of the

application domain and some creativity


Guidelines:

1. Interval of values specified by an input conditionEquivalent classes:

– Valid, for values that equal the interval’s boundaries and internal values “close” to the boundary

– Invalid, for values immediately below the minimum and above the maximum of the interval

2. Number of values specified by an input conditionEquivalent classes:

– Valid, for the minimum and maximum of the values and internal values “close” to the minimum and maximum

– Invalid, for numbers immediately below the minimum value and immediately above the maximum value


3. Repeat number 1. for each analogous output

condition

4. Repeat number 2. for each analogous output

condition

5. If input and/or output conditions include an

ordered set of elements you have to consider

also the first and the last elements of the set

6. Ask yourself if there are input conditions that

generate boundaries


• The output domain

• You have to carefully analyze the output domain considering that

– It is not always the case that boundaries of the input

domain correspond to the boundaries of the output

domain as they are determined by different

circumstances

– It is not always possible to generate a result out of

the boundaries of the output valid domain

Example of constraint

Requirements

Equivalence partitions

Valid Invalid

Sides are integers andthe sum of two facets isgreater than the thirdone

A + B > C A + B <=C

Boundary: A+B-1=C

Test cases

t1) A=4, B=4, C=6 : OK

t2) A=3, B=4, C=6 : OK

t3) A=3, B=2, C=6 : Error

[

C A+B-1

values

t3 t2 t1

BVA on IF THEN conditions

• We consider BVA on an If Then condition when it

contains disequations

• IF speed>50 THEN gear>2 (Valid partition)

• IF speed>50 THEN gear<=2 (Non valid partition)

– We negate ONLY the Then part !

• Valid partition:

– Minimum speed value that makes speed>50 TRUE : 51

– Minimum gear value that makes gear>2 TRUE: 3

65

BVA on IF THEN conditions

• Speed 50,51,52

• Gear 2,3,4

• Nine test cases error message!

• 50,2 50,3 50,4 51,2 51,3 51,4 52,2 52,3 52,4

66

x x x

x x x

Remarks

• Test cases designed by applying BVA catch very

common errors

• Random generation of test cases does not

identify such errors

• BVA is one of the most useful techniques in test

case design

• It seems easy, almost banal but it is not!

– Boundary conditions may be very subtle

software testing part i - dis.uniroma1.itsantucci/sw_engineering/material/05_a_test_parti.pdf ·...

Documents