software testing: introduction

Why? What? How?, When? This Course

Software Testing: Introduction

Mohammad Mousavi

Eindhoven University of Technology, The Netherlands

Software Testing, 2008

Mousavi: Software Testing: Introduction


Outline

Why?

What?

How?, When?

This Course



Why?

“Bugs”

I Facts of life! (correct by construction: notalways possible / affordable)

I Serious consequences (Pentium bug, OVChipcard, etc.) Ariane 5 explosion report:

This loss of information was due to specification anddesign errors in the software of the inertial referencesystem. ... The internal SRI software exception wascaused during execution of a data conversion from 64-bitfloating point to 16-bit signed integer value. The floatingpoint number which was converted had a value greaterthan what could be represented by a 16-bit signedinteger.



Why?

“Bugs”

I 2002 Costs: 60 Billion USD (only USA).I

Coders introduce bugs at the rateof 4.2 defects per hour ofprogramming. If you crack thewhip and force people to movemore quickly, things get evenworse. [Watts Humphreys]



Why?

Quest for QualityI

Software quality will become thedominant success criterion in thesoftware industry.

[ACM Workshop on StrategicDirections in Software Quality]

I Testing:I a way to achieve better qualityI >50% of the development costs



Outline

Why?

What?

How?, When?

This Course



What?

Sorts of “Bug”

I Error: conceptual mistakeI commission: implement the wrong

specificationI omission: forget to implement a

specification(the more difficult one to find and resolve)

I Fault/Bug/Defect: implemented error

I Failure: error shown in the behavior (Alsoanomaly, incident: observable failure.)



How?

Spec: A program that inputs an integer, and outputs 2 ∗ i3.

int i;i << cin;i = 2 * i;i = exp(i,3);cout << i;



How?Spec: A program that inputs an integer, and outputs 2 ∗ i3.

1: int i;2: i << cin;3: i = 2 * i;4: i = exp(i,3);5: cout << i;

I Error: confusing the binding power of operators

I Bug: Statements 3 and 4 are in the wrong order

I Fault:Test-case: on input 1, the program must output 2.input 1 ... output 8!



What?

Testing

I Planned experiments to:

1. reveal bugs (turn bugs into failures, test to fail),

Testing can show the presence of bugs, but not theabsence. [Dijkstra]

2. gain confidence in software (test to pass)



What?

I Test case (the plan):input, (behavior) and output

I Testing: planning and executing test-cases (how?).



Outline

Why?

What?

How?, When?

This Course



How?

Testing



How?

I Testing: planning and executing test-cases.

1. designing test-cases (manual, automatic: models, formalspecs),

2. executing them (manual or automatic: scaffolding, fixture),3. distinguishing failures or correct executions (manual: experts,

automatic: oracles, models)4. giving feed back to debug / change specification



How?

Aspects of Testing

I Functional testing:assumption: software is a function from inputs to outputscovering aspects of specificationsuitable for black-box testing (but can be enhanced withinformation from the code)

+ program independent: tests can be planned early+ tests are re-usable- gaps: untested pieces of software- redundancies: the same statements may be tested several times



Functional Testing: Mortgage Example

Spec. Write a program that takesthree inputs: gender (boolean), age([18-55]), salary ([0-10000])and output the total mortgage for one person

Mortgage = salary * factor,where factor is given by the following table.

Category Male FemaleYoung (18-35 years) 75 (18-30 years) 70Middle (36-45 years) 55 (31-40 years) 50Old (46-55 years) 30 (41-50 years) 35



An Implementation

Mortgage (male:Boolean, age:Integer, salary:Integer): Integerif male then

return ((18 ≤ age < 35)?(75 ∗ salary) : (31 ≤ age <40)?(55 ∗ salary) : (30 ∗ salary))

else {female}return ((18 ≤ age < 30)?(75 ∗ salary) : (31 ≤ age <40)?(50 ∗ salary) : (35 ∗ salary))

end if

Is this implementation correct?

No way, 12 bugs!



An Implementation




end if

Is this implementation correct? No way, 12 bugs!



Functional Testing




end if

Possible test cases:inputs: representatives from each age range and for each genderand salary 1,output: factors as given by the table(similar to equivalence testing; wait till next sessions!)



How?

Aspects of Testing

I Structural testing:covering aspects of programexamples: code coverage, branch coverage

+ giving insight to the effectiveness of test- more complicated than functional testing- incapable of detecting errors of omission



Structural Testing

Spec.: input: an integer x [1..216]output: x incremented by two, if x is less than 50,x decremented by one, if x is greater than 50, and50, otherwise.

if x < 50 thenx = x + 1;

end ifif x > 50 then

x = x - 1;end ifreturn x



Structural Testingif x < 50 then

x = x + 1;end ifif x > 50 then


Test-cases: sufficiently many random inputs until all statements areat least executed once, manually check the outputs with the spec.

Input Output Pass/Fail1540 1539 P2783 2782 P3222 3221 P30 31 F



Structural TestingFirst “Debugged” Version:

if x < 50 thenx = x + 2;



Input Output Pass/Fail1540 1539 P2783 2999 P3222 3221 P30 32 P

Have we tested enough?



Structural Testing

if x < 50 thenx = x + 2;



Input Output Pass/Fail49 50 F

Pesticide paradox: debugging old faults may produce new bugs (or“wake” old bugs up).



How?

Ideal Mix

I Functional testing for unit testing

I Structural testing for system-level testing

I Structural measures for the effectiveness of functionaltest-cases



When?

Waterfall Model



When?

Dealing with Bugs

1-3 Putting errors in (producing bugs),

4-6 finding bugs:I testingI fault classificationI fault isolation

7 removing bugs



What Else?

Alternatives

I Static Analysis:test abstract properties without running the program,e.g., uninitialized/unused variables, empty/unspecified cases,coding standards, checking for design (anti)patterns.

+ automatic and scalable for generic and abstract properties- involves approximation (true negatives and false positives);

complicated (may involve theorem proving) for concrete andspecific properties (proving the abstraction function to be“correct”)



What Else?

Alternatives

I Model Checking:test the state-space for formally specified properties.

+ rigorous analysis, push-button technology;- not (yet) applicable to many industrial cases (state-space

explosion)



When

Alternatives

I Model Checking: test all the state-space for formally specifiedproperties.

+ rigorous analysis, push-button technology;- not (yet) applicable to many industrial cases (state-space

explosion)



Outline

Why?

What?

How?, When?

This Course



Goals

I Learning about practical testing techniques (functional as wellas structural). The ability to choose among them based onthe assumptions of the problem in hand.

I Theoretical basis of testing. The ability to solve theoreticalproblems using the mathematical theory.

I Brief familiarity with debugging and automated debugging.Being able to apply debugging techniques.

I Getting an impression about Software Testing in practice.



Outline of This Course



General Information

I Home Page:http://www.win.tue.nl/∼mousavi/testing.htm

I Text book: Software Testing: A Craftsman Approach, 2ndEd., CRC Press, 2002. (3rd Ed. due in Feb. 2008)

I Marks:I Final exam on 25-06-2008, 09:00-12:00 (register before

03-06-2008).I Re-exam: 08-08-2008, 14:00-17:00.

I Office hours: Wednesdays, 14:30-15:30 or with priorappointment.


http://www.win.tue.nl/~mousavi/testing.htm


Acknowledgment

I have benefitted from the following resource in designing thiscourse:

I Judi Romijn’s material (slides, papers, handouts) for 2IW30

I Tim Willemse’s slides on LTS-based testing

I Other courses on Software Testing (named on the course webpage)


software testing: introduction

Documents

course mousavi

netherlands software

software qualitytesting

software industry

course outlinewhy

coursesoftware testing

error mousavi

wrong specication omission