CS/CE/SE 6367 Software Testing,

Validation and Verification

Lecture 01 Introduction

/42

Who am I?

• Instructor Name: Lingming Zhang • Office: ECSS 4.205 • Email: lingming.zhang@utdallas.edu • Homepage: http://www.utdallas.edu/~lxz144130/

2

/42

Education Background

3

/42

Industry Collaboration

4

/42

What am I doing?

5

/42

Hours and Resources      

• Course Meetings: • Tues/Thur 4:00pm – 5:15pm • GR 2.530

• Office Hours: • Tues/Thur 2:30pm - 3:30pm • ECSS 4.205

• Course Web Page: • http://www.utdallas.edu/~lxz144130/cs6367.html

       

6

/42

Required Textbook

7

/42

• Introduction to Software Testing (1st Edition) • ISBN: 978-0521880381

• Foundations of Software Testing (2nd Edition) • ISBN: 978-8131794760

Recommended Textbooks

8

/42

Grading Scheme       

30

1020

20

20

Exam1Exam2HomeworkQuiz&Class ParticipationCourse Project

5

10

15

ImplementationReportPresentation

Overall Course Project

9

/42

Grading Scale

Score Grade93-100 A90-92 A-87-89 B+83-86 B80-82 B-77-79 C+70-76 C<70 F

10

I may choose to curve the grades at the end of the term

/42

More on the Course Project

• A research project: chosen from a set of topics (posted later), or proposed by the students • Analyzing/testing real-world Java code

• Work individually or in pair (1-2 people) • Go through the whole research project process

• Proposal • Implementation&Experimentation • Report writing • Project presentation

The way to learn software engineering is to go out there and do software engineering. 

-- Fred Brooks

11

/42

Project Outcomes

• 1-2 page project proposal [Due by Feb 29th midnight] • Source code [Due by April 30th midnight] • Project report in ACM SIGPLAN conference format (5-10

pages, double column) [Due by April 30th midnight] • Final project presentation&Demo

12

/42

Now, let’s start!

13

/42

Software is Everywhere

14

/42

Software Characteristics

Complexity Evolution

15

/42

The Debian OS Example

Year OS LoC(Million)

2000 Debian2.2 55-59

2002 Debian3.0 104

2005 Debian3.1 215

2007 Debian4.0 283

2009 Debian5.0 324

2012 Debian7.0 419

Complexity

Evolution

16

/42

The Facts• Only 32% of software projects are considered successful

(full featured, on time, on budget) • Software failures cost the US economy $59.5 billion

dollars every year [NIST 2002 Report] • On average, 1-5 bugs per KLOC (thousand lines of code)

• In mature software (more than 10 bugs in prototypes)

✴ 35MLOC ✴ 63K known bugs at the time of release ✴ 2 bugs per KLOC

17

/42

Software Fault Examples

• Pac-Man (1980) • Should always have no ending • Has “Split Screen” at level 256

• Cause: Integer overflow • 8 bits: maximum representable value 28 − 1 = 255

+ =1 1 1 1 1 1 1 1

2550 0 0 0 0 0 0 1

11 0 0 0 0 0 0 0 0

0

18

/42

Software Fault Examples

• Mars Climate Orbiter (1998) • Sent to Mars to relay signal from Mars

Lander • Smashed to the planet

• Cause: Failing to convert between different metric standards • Software that calculated the total impulse

presented results in pound-seconds • The system using these results expected

its inputs to be in newton-seconds

19

/42

Software Fault Examples

• USS Yorktown (1997) • Left dead in the water for 3 hours

• Cause: Divide by zero error

Number0 =

20

/42

Software Fault Examples

• THERAC-25 Radiation Therapy (1985) • 2 cancer patients received fatal

overdoses • Cause:

• Miss-handling of race condition of the software in the equipment

21

/42

Software Fault Examples

• ATT (1990) • One switching system in New

York City experienced an intermittent failure that caused a major service outage

• The first major network problem in AT&T’s 114-year history

• Cause: Wrong BREAK statement in C Code • Complete code coverage could

have revealed this bug during testing

1. network  code()    2. {    3. switch  (line)  {    4.        case  THING1:    5.                doit1();    6.                break;    7.  case  THING2:    8.                  if  (x  ==  STUFF)  {    9.                        do_first_stuff();    10.                        if  (y  ==  OTHER_STUFF)    11.                                  break;    12.                        do_later_stuff();}    13.  /*  coder  meant  to  break  to  here...  */    14.                    initialize_modes_pointer();    15.                    break;    16.        default:    17.                  processing();  }  18.  /*  ...but  actually  broke  to  here!  */    19.       u s e _ m o d e s _ p o i n t e r ( ) ; /

*  leaving  the  modes_pointer  20.        uninitialized  */    21.}    

22

/42

Software Fault Examples

• Ariane 5 flight 501 (1996) • Destroyed 37 seconds after

launch (cost: $370M) • Cause: Arithmetic overflow

• Data conversion from a 64-bit floating point to 16-bit signed integer value caused an exception

• The software from Ariane 4 was re-used for Ariane 5 without re-testing

23

/42

Software Failure, Fault & Error

• Fault • Incorrect portions of code (may involve missing code as well

as incorrect code) • Necessary (not sufficient) condition for the occurrence of a

failure • Failure

• Observable incorrect behavior of a program. • Error

• Cause of a fault. something bad a programmer did (conceptual, typo, etc)

• Bug: informal term for fault/failure

24

/42

Approaches to reduce faults

• Manual code review • Manually review the code to detect faults • Limitations:

• Hard to evaluate your progress • Can miss many faults/bugs

25

/42

Approaches to reduce faults

• Manual code review

26

/42

Automated approaches to reduce faults

Static Analysis Testing Verification

27

/42

Automated approaches to reduce faults

Static Analysis Testing Verification

• Static analysis: Identify specific problems (e.g., memory leak) in the software by scanning suspicious patterns from the code • Limitations: (1) Limited problem types, (2) False positives

28

/42

Automated approaches to reduce faults

Static Analysis Testing Verification

• Testing: Feed input to software and run it to see whether its behavior is as expected

• Limitations: (1) Impossible to cover all possible execution, (2) Need test oracles

29

/42

Automated approaches to reduce faults

Static Analysis Testing Verification

• Formal Verification: Consider all the possible program executions, and formally prove that the program is correct or not

• Limitations: (1) Difficult to have a formal specification, (2) Most real-world programs are too expensive to prove

30

/42

The Most Widely Used Approach

31

“50% of my employees are testers, and the rest spends 50% of their time testing”

Testing!

/42

Why Testing?

• Testing vs. code review: • More reliable than code review

• Testing vs. static checking: • Less false positive and applicable to more problems

• Testing vs. formal verification: • More scalable and applicable to more programs

• You get what you pay (linear rewards) • While the others are not!

32

/42

What will this course cover?

program test inputs outputs+

expectedoutputs

=?

output?=expected output

Testing Small Code

int sum(int x, int y){return x-y;//bug:x+y

}

x=1, y=0

x=1, y=2

x=2, y=2

x=1, y=0

x=1, y=2

x=2, y=2

33

/42

What will this course cover?

program test inputs outputs+

expectedoutputs

=?

layout?=expected layoutTesting Browsers

34

/42

What will this course cover?

compiled code?=expected codeTesting Compilers

program test inputs outputs+

expectedoutputs

=?

35

/42

What will this course cover?

program test inputs outputs+

expectedoutputs

=?

• How to analyze program source code to find potential software faults? [Software Analysis]

• How to formally verify program to find all the possible software faults? [Software Verification/Formal Methods]

36

/42

What will this course cover?

• How to evaluate the quality of generated tests? [Structural Code Coverage, Mutation Testing]

• How to generate high-quality tests automatically? [Automated Test Generation]

• How to run tests faster when program evolves? [Regression Testing]

program test inputs outputs+

expectedoutputs

=?

37

/42

program test inputs outputs+

expectedoutputs

=?

• How to automatically generate test oracle/assertions? [Oracle Generation]

• How to automatically localize program faults? [Automated Debugging]

• How to automatically fix program faults? [Automated Program Repair]

test oracle/assertion

What will this course cover?

38

/42

What else will this course cover?

• How to analyze Java source code? • E.g., using Eclipse JDT

• How to analyze and modify Java bytecode? • E.g., using ASM Java bytecode analysis framework

• How to analyze and modify Java intermediate code representation? • E.g., using WALA/Soot

39

/42

Course Topics• Basic concepts • Test adequacy • Automated test generation • Mutation Testing • Regression testing • Automated oracle generation • Automated debugging • Automated program repair • Software analysis • Formal methods in software testing

40

/42

Why This Course?

• In academia • Advance your current research • Find your future research interests

• In industry • QAs/Software Testers/Test Engineers are in high

demand • Software engineers are also strongly related to software

testing/verification

41

/42

Thanks! Hope you will enjoy the course!

42