software engineering - lecture 11: testing and debugging ... · motivation debugging...
TRANSCRIPT
![Page 1: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/1.jpg)
Software EngineeringLecture 11: Testing and Debugging — Debugging
Peter Thiemann
University of Freiburg, Germany
10.06.2013
![Page 2: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/2.jpg)
Motivation
Debugging is unavoidable and a major economical factor
I Software bugs cost the US economy ca. 60 billion US$/y(2002)In general estimated 0.6% of the GDP of industrial countries
I Ca. 80 percent of software development costs spent onidentifying and correcting defects
I Software re-use is increasing and tends to introduce bugsdue to changed specifications in new context (Ariane 5)
Debugging needs to be systematic
I Bug reports may involve large inputs
I Programs may have thousands of memory locations
I Programs may pass through millions of states before failureoccurs
![Page 3: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/3.jpg)
Motivation
Debugging is unavoidable and a major economical factor
I Software bugs cost the US economy ca. 60 billion US$/y(2002)In general estimated 0.6% of the GDP of industrial countries
I Ca. 80 percent of software development costs spent onidentifying and correcting defects
I Software re-use is increasing and tends to introduce bugsdue to changed specifications in new context (Ariane 5)
Debugging needs to be systematic
I Bug reports may involve large inputs
I Programs may have thousands of memory locations
I Programs may pass through millions of states before failureoccurs
![Page 4: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/4.jpg)
Terminology
Bug-Related Terminology
1. Defect (aka bug, fault) introduced to the code by programmerNot always programmer’s fault: changing/unforeseenrequirements
2. Defect may cause infection of the program state duringexecutionNot all defects cause an infection: e.g., Pentium bug
3. An infected state propagates during executionInfected parts of states may be overwritten, corrected, unused
4. Infection may cause a failure: externally observable errorMay include non-termination
Defect — Infection — Propagation — Failure
![Page 5: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/5.jpg)
The Main Steps in Systematic Debugging
Program State
Tim
e
4
8
8
earliest state known to be infected
latest state known to be healthy
Reproduce failure with test input
![Page 6: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/6.jpg)
The Main Steps in Systematic Debugging
Program State
Tim
e
4
8
8
earliest state known to be infected
latest state known to be healthy
Reduction of failure-inducing problem
![Page 7: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/7.jpg)
The Main Steps in Systematic Debugging
Program State
Tim
e
4
8
8
earliest state known to be infected
latest state known to be healthy
State known to be healthy
![Page 8: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/8.jpg)
The Main Steps in Systematic Debugging
Program State
Tim
e
4
8
8
earliest state known to be infected
latest state known to be healthy
State known to be infected
![Page 9: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/9.jpg)
The Main Steps in Systematic Debugging
Program State
Tim
e
4
8
8
earliest state known to be infected
latest state known to be healthy
State where failure becomes observable
![Page 10: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/10.jpg)
The Main Steps in Systematic Debugging
Program State
Tim
e
4
8
8
earliest state known to be infected
latest state known to be healthy
I Separate healthy from infected states
![Page 11: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/11.jpg)
The Main Steps in Systematic Debugging
Program State
Tim
e
4
8
8
earliest state known to be infected
latest state known to be healthy
I Separate healthy from infected states
I Separate relevant parts from irrelevant ones
![Page 12: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/12.jpg)
Debugging Techniques
The analysis suggests main techniques used in systematicdebugging:
I Bug tracking — Which initial states cause failure?
I Program control — Design for Debugging
I Input simplification — Reduce state space
I State observation and watching using debuggers
I Tracking causes and effects — From failure to defect
Common ThemesI Fighting combinatorial explosion: separate relevant from
irrelevant
I Being systematic: avoid repetition, ensure progress, use tools
![Page 13: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/13.jpg)
Debugging Techniques
The analysis suggests main techniques used in systematicdebugging:
I Bug tracking — Which initial states cause failure?
I Program control — Design for Debugging
I Input simplification — Reduce state space
I State observation and watching using debuggers
I Tracking causes and effects — From failure to defect
Common ThemesI Fighting combinatorial explosion: separate relevant from
irrelevant
I Being systematic: avoid repetition, ensure progress, use tools
![Page 14: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/14.jpg)
Bug Tracking
![Page 15: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/15.jpg)
Bug Tracking Life Cycle
Unconfirmed
New Assigned Resolved
Invalid
Duplicate
Verified Closed
Reopened
Raw problem report, often from end user
![Page 16: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/16.jpg)
Bug Tracking Life Cycle
Unconfirmed New
Assigned Resolved
Invalid
Duplicate
Verified Closed
Reopened
Description complete, no duplicate of existing bug
![Page 17: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/17.jpg)
Bug Tracking Life Cycle
Unconfirmed New Assigned
Resolved
Invalid
Duplicate
Verified Closed
Reopened
![Page 18: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/18.jpg)
Bug Tracking Life Cycle
Unconfirmed New Assigned Resolved
Invalid
Duplicate
Verified Closed
Reopened
Tag MeaningFixed Problem is fixed
WontFix Impossible or undesirable (“feature”) to fixWorksForMe Can’t be reproduced
Invalid Not a problem or insufficiently describedDuplicate Refers to existing problem
![Page 19: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/19.jpg)
Bug Tracking Life Cycle
Unconfirmed New Assigned Resolved
Invalid
Duplicate
Verified Closed
Reopened
![Page 20: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/20.jpg)
Bug Tracking Life Cycle
Unconfirmed New Assigned Resolved
Invalid
Duplicate
Verified
Closed
Reopened
Only if Fixed
![Page 21: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/21.jpg)
Bug Tracking Life Cycle
Unconfirmed New Assigned Resolved
Invalid
Duplicate
Verified Closed
Reopened
Important to avoid cluttering of bug database
![Page 22: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/22.jpg)
Bug Tracking Life Cycle
Unconfirmed New Assigned Resolved
Invalid
Duplicate
Verified Closed
Reopened
The fix didn’t work after all . . .
![Page 23: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/23.jpg)
Bugzilla’s Bug Lifecycle
![Page 24: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/24.jpg)
From Bug to Test Case
![Page 25: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/25.jpg)
Program Control: From Bug to Test Case
Scenario
Assume Firefox crashes while printing a certain URL to file
We need to turn the bug report into an automated test case!
Automated test case execution essentialI Reproduce the bug reliably (cf. scientific experiment)
I Repeated execution necessary during isolation of defect
I After successful fix, test case becomes part of test suite
Prerequisites for automated execution
1. Program control (without manual interaction)
2. Isolating small program units that contain the bug
![Page 26: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/26.jpg)
Program Control: From Bug to Test Case
Scenario
Assume Firefox crashes while printing a certain URL to file
We need to turn the bug report into an automated test case!
Automated test case execution essentialI Reproduce the bug reliably (cf. scientific experiment)
I Repeated execution necessary during isolation of defect
I After successful fix, test case becomes part of test suite
Prerequisites for automated execution
1. Program control (without manual interaction)
2. Isolating small program units that contain the bug
![Page 27: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/27.jpg)
Program Control
![Page 28: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/28.jpg)
Program Control
Enable automated run of program that may involve userinteraction
Example (Sequence of interaction that led to the crash)
1. Launch Firefox
2. Open URL location dialogue
3. Type in a location
4. Open Print dialogue
5. Enter printer settings
6. Initiate printing
![Page 29: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/29.jpg)
Alternative Program Interfaces for Testing
Presentation Layer
(Common) Functionality Layer
Unit Unit Unit
User
Automated Test
![Page 30: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/30.jpg)
Alternative Program Interfaces for Testing
Presentation Layer
(Common) Functionality Layer
Unit Unit Unit
User Automated Test
![Page 31: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/31.jpg)
Automated Testing at Different Layers
Presentation
Scripting languages for capturing & replaying user I/O
I Specific to an OS/Window system/Hardware
I Scripts tend to be brittle
Functionality
Interface scripting languages
1. Implementation-specific scripting languages: VBScript
2. Universal scripting languages withapplication-specific extension: Python, Perl, Tcl
Unit testing frameworks (as in previous lecture)
JUnit, CPPUnit, VBUnit, . . .
![Page 32: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/32.jpg)
Testing Layers: Discussion
The higher the layer, the more difficult becomes automated testing
I Scripting languages specific to OS/Window S./Progr. L.I Test scripts depend on (for example):
I application environment (printer driver)I hardware (screen size), working environment (paper size)
Test at the unit layer whenever possible!
Requires modular design with low coupling
I Good design is essential even for testing and debugging!
I We concentrate on decoupling rather than specific scripts
![Page 33: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/33.jpg)
Testing Layers: Discussion
The higher the layer, the more difficult becomes automated testing
I Scripting languages specific to OS/Window S./Progr. L.I Test scripts depend on (for example):
I application environment (printer driver)I hardware (screen size), working environment (paper size)
Test at the unit layer whenever possible!
Requires modular design with low coupling
I Good design is essential even for testing and debugging!
I We concentrate on decoupling rather than specific scripts
![Page 34: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/34.jpg)
Testing Layers: Discussion
The higher the layer, the more difficult becomes automated testing
I Scripting languages specific to OS/Window S./Progr. L.I Test scripts depend on (for example):
I application environment (printer driver)I hardware (screen size), working environment (paper size)
Test at the unit layer whenever possible!
Requires modular design with low coupling
I Good design is essential even for testing and debugging!
I We concentrate on decoupling rather than specific scripts
![Page 35: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/35.jpg)
Disentangling Layers
Core+print to file()
UserPresentation+confirm loss()
invokes
invokes
Circular Dependency Example
I Print-to-file is core functionalitycalls confirm loss() to prevent accidental file removal
I Override-if-file-exists question is in UIrelies on core functionality to check file existence
![Page 36: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/36.jpg)
Breaking Circular Dependencies by Refactoring
Core+print to file()
Presentation+confirm loss()
UserPresentation+confirm loss()
TestPresentation+confirm loss()
invokes
invokes
return true;
I Programming to interfaces important even for testability
![Page 37: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/37.jpg)
Isolating Units
Use test interfaces to isolate smallest unit containing the defect
I In the Firefox example, unit for file printing easily identified
I In general, use debugger to trace execution
![Page 38: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/38.jpg)
Problem Simplification
![Page 39: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/39.jpg)
From Bug to Test Case, Part II
Scenario
Assume Firefox crashes while printing a loaded URL to file
We need to turn the bug report into an automated test case!
We managed to isolate the relevant program unit, but . . .
<!DOCTYPE HTML PUBLIC " -//W3C//DTD HTML 4.01// EN
">
<html lang="en">
<head>
<title>Mozilla.org</title>
<meta http -equiv="Content -Type"
content="text/html; charset=UTF -8">
... about 200 lines more
![Page 40: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/40.jpg)
From Bug to Test Case, Part II
Scenario
Assume Firefox crashes while printing a loaded URL to file
We need to turn the bug report into an automated test case!
We managed to isolate the relevant program unit, but . . .
<!DOCTYPE HTML PUBLIC " -//W3C//DTD HTML 4.01// EN
">
<html lang="en">
<head>
<title>Mozilla.org</title>
<meta http -equiv="Content -Type"
content="text/html; charset=UTF -8">
... about 200 lines more
![Page 41: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/41.jpg)
Problem Simplification
We need a small test case that fails!
Divide-and-Conquer
1. Cut away one half of the test input
2. Check, whether one of the halves still exhibits failure
3. Continue until minimal failing input is obtained
ProblemsI Tedious: rerun tests manually
I Boring: cut-and-paste, rerun
I What, if none of the halves exhibits a failure?
![Page 42: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/42.jpg)
Problem Simplification
We need a small test case that fails!
Divide-and-Conquer
1. Cut away one half of the test input
2. Check, whether one of the halves still exhibits failure
3. Continue until minimal failing input is obtained
ProblemsI Tedious: rerun tests manually
I Boring: cut-and-paste, rerun
I What, if none of the halves exhibits a failure?
![Page 43: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/43.jpg)
Problem Simplification
We need a small test case that fails!
Divide-and-Conquer
1. Cut away one half of the test input
2. Check, whether one of the halves still exhibits failure
3. Continue until minimal failing input is obtained
ProblemsI Tedious: rerun tests manually
I Boring: cut-and-paste, rerun
I What, if none of the halves exhibits a failure?
![Page 44: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/44.jpg)
Automatic Input Simplification
I Automate cut-and-paste and re-running tests
I Increase granularity of chunks when no failure occurs
Example
public static int checkSum(int[] a)
I is supposed to compute the checksum of an integer array
I gives wrong result, whenever a contains two identicalconsecutive numbers, but we don’t know that yet
I we have a failed test case, eg, from transmission trace:
{1,3,5,3,9,17,44,3,6,1,1,0,44,1,44,0}
![Page 45: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/45.jpg)
Automatic Input Simplification
I Automate cut-and-paste and re-running tests
I Increase granularity of chunks when no failure occurs
Example
public static int checkSum(int[] a)
I is supposed to compute the checksum of an integer array
I gives wrong result, whenever a contains two identicalconsecutive numbers, but we don’t know that yet
I we have a failed test case, eg, from transmission trace:
{1,3,5,3,9,17,44,3,6,1,1,0,44,1,44,0}
![Page 46: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/46.jpg)
Input Simplification (n = number of chunks)
1 3 5 3 9 17 44 3 6 1 1 0 44 1 44 0 8
n=2
1 3 5 3 9 17 44 3 4
6 1 1 0 44 1 44 0 8
6 1 1 0 8
6 1 4
1 0 4
n=4 increase granularity
6 1 1 8
n=3 adjust granularity to input size
6 1 4
1 1 8
![Page 47: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/47.jpg)
Input Simplification (n = number of chunks)
1 3 5 3 9 17 44 3 6 1 1 0 44 1 44 0 8
n=2
1 3 5 3 9 17 44 3 4
6 1 1 0 44 1 44 0 8
6 1 1 0 8
6 1 4
1 0 4
n=4 increase granularity
6 1 1 8
n=3 adjust granularity to input size
6 1 4
1 1 8
![Page 48: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/48.jpg)
Input Simplification (n = number of chunks)
1 3 5 3 9 17 44 3 6 1 1 0 44 1 44 0 8
n=2
1 3 5 3 9 17 44 3 4
6 1 1 0 44 1 44 0 8
6 1 1 0 8
6 1 4
1 0 4
n=4 increase granularity
6 1 1 8
n=3 adjust granularity to input size
6 1 4
1 1 8
![Page 49: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/49.jpg)
Input Simplification (n = number of chunks)
1 3 5 3 9 17 44 3 6 1 1 0 44 1 44 0 8
n=2
1 3 5 3 9 17 44 3 4
6 1 1 0 44 1 44 0 8
6 1 1 0 8
6 1 4
1 0 4
n=4 increase granularity
6 1 1 8
n=3 adjust granularity to input size
6 1 4
1 1 8
![Page 50: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/50.jpg)
Input Simplification (n = number of chunks)
1 3 5 3 9 17 44 3 6 1 1 0 44 1 44 0 8
n=2
1 3 5 3 9 17 44 3 4
6 1 1 0 44 1 44 0 8
6 1 1 0 8
6 1 4
1 0 4
n=4 increase granularity
6 1 1 8
n=3 adjust granularity to input size
6 1 4
1 1 8
![Page 51: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/51.jpg)
Input Simplification (n = number of chunks)
1 3 5 3 9 17 44 3 6 1 1 0 44 1 44 0 8
n=2
1 3 5 3 9 17 44 3 4
6 1 1 0 44 1 44 0 8
6 1 1 0 8
6 1 4
1 0 4
n=4 increase granularity
6 1 1 8
n=3 adjust granularity to input size
6 1 4
1 1 8
![Page 52: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/52.jpg)
Input Simplification (n = number of chunks)
1 3 5 3 9 17 44 3 6 1 1 0 44 1 44 0 8
n=2
1 3 5 3 9 17 44 3 4
6 1 1 0 44 1 44 0 8
6 1 1 0 8
6 1 4
1 0 4
n=4 increase granularity
6 1 1 8
n=3 adjust granularity to input size
6 1 4
1 1 8
![Page 53: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/53.jpg)
Simplification Algorithm — Delta Debugging
Prerequisites
I test(c) ∈ {4,8, ?} runs a test on configuration cI Let c8 be a failing input configuration with
I test(c8) = 8I length l = |c8| if c8 = {x1, . . . , xl}I view at granularity n ≤ l : c8 = c1 ∪ · · · ∪ cn, ci 6= ∅I write ci ∈n c
Find minimal failing input: call ddMin(c0, 2) with test(c0) = 8
ddMin(c8, n) =c8 |c8| = 1
ddMin(c8 − c ,max(n−1, 2)) c ∈n c8 ∧ test(c8 − c) = 8
ddMin(c8,min(2n, |c8|)) n < |c8|c8 otherwise
![Page 54: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/54.jpg)
Simplification Algorithm — Delta Debugging
Prerequisites
I test(c) ∈ {4,8, ?} runs a test on configuration cI Let c8 be a failing input configuration with
I test(c8) = 8I length l = |c8| if c8 = {x1, . . . , xl}I view at granularity n ≤ l : c8 = c1 ∪ · · · ∪ cn, ci 6= ∅I write ci ∈n c
Find minimal failing input: call ddMin(c0, 2) with test(c0) = 8
ddMin(c8, n) =c8 |c8| = 1
ddMin(c8 − c ,max(n−1, 2)) c ∈n c8 ∧ test(c8 − c) = 8
ddMin(c8,min(2n, |c8|)) n < |c8|c8 otherwise
![Page 55: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/55.jpg)
Minimal Failure Configuration
I Minimization algorithm is easy to implement
I Realizes input size minimization for failed runI Implementation:
I Small program in your favorite PL (Zeller: Python, Java)I Eclipse plugin DDinput at
www.st.cs.uni-sb.de/eclipse/
I Demo: DD.java, Dubbel.java
Consequences of Minimization
I Input small enough for observing, tracking, locating (nexttopics)
I Minimal input often provides important hint for source ofdefect
![Page 56: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/56.jpg)
Minimal Failure Configuration
I Minimization algorithm is easy to implement
I Realizes input size minimization for failed runI Implementation:
I Small program in your favorite PL (Zeller: Python, Java)I Eclipse plugin DDinput at
www.st.cs.uni-sb.de/eclipse/
I Demo: DD.java, Dubbel.java
Consequences of Minimization
I Input small enough for observing, tracking, locating (nexttopics)
I Minimal input often provides important hint for source ofdefect
![Page 57: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/57.jpg)
Principal Limitations of Input Minimization
I Algorithm computes minimal failure-inducting subsequence ofthe input:Taking away any chunk of any length removes the failure
I However, there may be failing inputs with smaller size!
1. Algorithm investigates only one failing input of smaller size2. Misses failure-inducing inputs created by taking away several
chunks
Example (Incompleteness of minimization)
Failure occurs for integer array when frequency of occurrences ofall numbers is even:
{1,2,1,2} failsTaking away any chunk of size 1 or 2 passes{1,1} fails, too, and is even smaller
![Page 58: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/58.jpg)
Principal Limitations of Input Minimization
I Algorithm computes minimal failure-inducting subsequence ofthe input:Taking away any chunk of any length removes the failure
I However, there may be failing inputs with smaller size!
1. Algorithm investigates only one failing input of smaller size2. Misses failure-inducing inputs created by taking away several
chunks
Example (Incompleteness of minimization)
Failure occurs for integer array when frequency of occurrences ofall numbers is even:
{1,2,1,2} failsTaking away any chunk of size 1 or 2 passes{1,1} fails, too, and is even smaller
![Page 59: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/59.jpg)
Principal Limitations of Input Minimization
I Algorithm computes minimal failure-inducting subsequence ofthe input:Taking away any chunk of any length removes the failure
I However, there may be failing inputs with smaller size!
1. Algorithm investigates only one failing input of smaller size2. Misses failure-inducing inputs created by taking away several
chunks
Example (Incompleteness of minimization)
Failure occurs for integer array when frequency of occurrences ofall numbers is even:
{1,2,1,2} fails
Taking away any chunk of size 1 or 2 passes{1,1} fails, too, and is even smaller
![Page 60: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/60.jpg)
Principal Limitations of Input Minimization
I Algorithm computes minimal failure-inducting subsequence ofthe input:Taking away any chunk of any length removes the failure
I However, there may be failing inputs with smaller size!
1. Algorithm investigates only one failing input of smaller size2. Misses failure-inducing inputs created by taking away several
chunks
Example (Incompleteness of minimization)
Failure occurs for integer array when frequency of occurrences ofall numbers is even:
{1,2,1,2} failsTaking away any chunk of size 1 or 2 passes
{1,1} fails, too, and is even smaller
![Page 61: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/61.jpg)
Principal Limitations of Input Minimization
I Algorithm computes minimal failure-inducting subsequence ofthe input:Taking away any chunk of any length removes the failure
I However, there may be failing inputs with smaller size!
1. Algorithm investigates only one failing input of smaller size2. Misses failure-inducing inputs created by taking away several
chunks
Example (Incompleteness of minimization)
Failure occurs for integer array when frequency of occurrences ofall numbers is even:
{1,2,1,2} failsTaking away any chunk of size 1 or 2 passes{1,1} fails, too, and is even smaller
![Page 62: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/62.jpg)
Limitations of Linear Minimization
Minimization algorithm ignores structure of input
Example (.html input configuration)
<SELECT NAME="priority"MULTIPLE SIZE=7> 8
I Most substrings are not valid HTML: test result ?(“unresolved”)
I There is no point to test beneath granularity of tokens
Minimization may require a very large number of steps
![Page 63: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/63.jpg)
Structured Minimization
SELECT
NAME
"priority"
MULTIPLE SIZE
7
0
1
1.1
2 3
3.1
Linearization of c8:
<SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7>
Input configuration consists of nodes in ABS not characters
![Page 64: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/64.jpg)
Structured Minimization
SELECT
NAME
"priority"
MULTIPLE SIZE
7
0
1
1.1
2 3
3.1
Linearization of c8:<SELECT NAME="priority" MULTIPLE SIZE=7>
<SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7>
c8 = {0, 1, 1.1, 2, 3, 3.1}
![Page 65: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/65.jpg)
Structured Minimization
SELECT
NAME
"priority"
MULTIPLE SIZE
7
0
1
1.1
2 3
3.1
Linearization of c8:<SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7>
<SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7>
c8 = {0, 1, 1.1, 2, 3, 3.1} infeasible (not a tree) return ?
![Page 66: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/66.jpg)
Structured Minimization
SELECT
NAME
"priority"
MULTIPLE SIZE
7
0
1
1.1
2 3
3.1
Linearization of c8:<SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7>
<SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7>
c8 = {0, 1, 1.1, 2, 3, 3.1} Failure occurs, reduce length
![Page 67: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/67.jpg)
Structured Minimization
SELECT
NAME
"priority"
MULTIPLE SIZE
7
0
1
1.1
2 3
3.1
Linearization of c8:<SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7>
<SELECT NAME="priority" MULTIPLE SIZE=7>
c8 = {0, 1, 1.1, 2, 3, 3.1} infeasible (not well-formed HMTL) return ?
![Page 68: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/68.jpg)
Structured Minimization
SELECT
NAME
"priority"
MULTIPLE SIZE
7
0
1
1.1
2 3
3.1
Linearization of c8:<SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7><SELECT NAME="priority" MULTIPLE SIZE=7>
c8 = {0, 1, 1.1, 2, 3, 3.1} Failure occurs, can’t be minimized further
![Page 69: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/69.jpg)
Delta Debugging, Adaptive Testing
The Bigger Picture
I Minimization of failure-inducing input is instance of deltadebugging
I Delta debugging is instance of adaptive testing
Definition (Delta Debugging)
Isolating failure causes by narrowing down differences (“δ”)between runs
This principle is used in various debugging activities
Definition (Adaptive Testing)
Test series where each test depends on the outcome of earlier tests
![Page 70: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/70.jpg)
Delta Debugging, Adaptive Testing
The Bigger Picture
I Minimization of failure-inducing input is instance of deltadebugging
I Delta debugging is instance of adaptive testing
Definition (Delta Debugging)
Isolating failure causes by narrowing down differences (“δ”)between runs
This principle is used in various debugging activities
Definition (Adaptive Testing)
Test series where each test depends on the outcome of earlier tests
![Page 71: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/71.jpg)
Delta Debugging, Adaptive Testing
The Bigger Picture
I Minimization of failure-inducing input is instance of deltadebugging
I Delta debugging is instance of adaptive testing
Definition (Delta Debugging)
Isolating failure causes by narrowing down differences (“δ”)between runs
This principle is used in various debugging activities
Definition (Adaptive Testing)
Test series where each test depends on the outcome of earlier tests
![Page 72: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/72.jpg)
What Next?
4 Bug tracking
4 Program control — Design for Debugging
4 Input simplification
I Execution observationI With loggingI Using debuggers
I Tracking causes and effects
![Page 73: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/73.jpg)
What Next?
4 Bug tracking
4 Program control — Design for Debugging
4 Input simplification
I Execution observationI With loggingI Using debuggers
I Tracking causes and effects
![Page 74: Software Engineering - Lecture 11: Testing and Debugging ... · Motivation Debugging isunavoidableand a majoreconomical factor I Software bugs cost the US economy ca. 60 billion US$/y](https://reader035.vdocument.in/reader035/viewer/2022070809/5f07bc5b7e708231d41e7af3/html5/thumbnails/74.jpg)
Literature for this Lecture
Essential
Zeller Why Programs Fail: A Guide to SystematicDebugging, Morgan Kaufmann, 2005Chapters 2, 3, 5
Background
McConnell Code Complete: A Practical Handbook for SoftwareConstruction, 2nd edition, Microsoft Press, 2004Chapter 23