How Failures Come To BeBy: Blake Burton

Golden rule:Software testing begins with careful design

• Modularity of your code is a big part in software testing– Design your program to be comprised of small modules or

functions– It is easier to do testing/tracing with small functions.

• When the program is made up of small functions and something goes wrong the bug can be traced to a SINGLE function or a SMALL group of functions.

• It is easier to test several small functions comprised of 20 lines or so than one block of 100+ lines

– Example• You have a program that is a calculator. You notice that the modulo

function is returning the incorrect numbers.• It is easy to go to a function that takes care of the modulo function

and debug that single function rather than the whole program

Golden Rule:Software that cannot be tested in useless

• Imagine you have a large enterprise application. If this application is poorly designed (ex/ all code in one file under main). This becomes almost impossible to test the program to see if something is wrong. What is the purpose of a program that cannot be tested? If a program cannot be tested it may have bugs/defects. If these bugs go into development and crash the application, this application is useless.

Facts On Debugging: Famous Failure

• Night Capital Group, a market-making firm that until August 2012 had a stellar reputation in its industry, blew all of that in about 30 minutes. Between 9:30 a.m. and 10 a.m. EST on August 1, the company’s trading algorithms got a little buggy and decided to buy high and sell low on 150 different stocks. http://www.intertech.com/Blog/15-worst-computer-software-blunders/

Facts On Debugging: damages Caused by Software Bugs

• By the time the bleeding had stopped, KCG had lost $440 million on trades. By comparison, its market cap is just $296 million and the loss was four times its 2011 net income. Furthermore, the company’s stock price drop 62% in one day, according to Bloomberg Businessweek.

• http://www.intertech.com/Blog/15-worst-computer-software-blunders/

Facts On Debugging: damages Caused by Software Bugs

• Cambridge University Study States Software Bugs Cost Economy $312 Billion Per Year

• http://www.prweb.com/releases/2013/1/prweb10298185.htm

Facts About Debugging

• Reworking defects in requirements, design, and code consumes 40-50% of the total cost of software development. [1]

• Every hour spent on defect prevention will reduce repair time by 3-10 hours. [2]

• Reworking a requirements problem once the software has shipped costs 50-200 times what it would take to rework during requirements. [3]

• 60% of all defects usually exist at design time. [4]1. (Boehm, Barry W. 1987. "Improving Software Productivity." IEEE Computer, September: 43-57.; Jones, Capers, ed. 1986.

Tutorial: Programming Productivity: Issues for the Eighties, 2nd ed. Los Angeles: IEEE Computer Society Press.)

2. (Jones, Capers. 1994. Assessment and Control of Software Risks. Englewood Cliffs, NJ: Yourdon Press.)

3. (Boehm, Barry W., and Philip N. Papaccio. 1988. "Understanding and Controlling Software Costs." IEEE Transactions on Software Engineering, vol. 14, no. 10 (October): 1462-1477.)

4. (Glib, Tom. 1988. Principles of Software Engineering Management. Wokingham, England: Addison-Wesley.)

5. All facts grabbed from http://programmers.stackexchange.com/questions/91758/debugging-facts-and-statistics

Sample Main

int main(int argc, char *argv[]){ int *a; int I;

a = (int *)malloc((argc - 1) * sizeof(int)); for (i = 0; i < argc - 1; i++) a[i] = atoi(argv[i + 1]);

shell_sort(a, argc);

printf("Output: "); for (i = 0; i < argc - 1; i++) printf("%d ", a[i]); printf("\n");

free(a); return 0;}

Sample Sort

static void shell_sort(int a[], int size){ int i, j; int h = 1; do { h = h * 3 + 1; } while (h <= size); do { h /= 3; for (i = h; i < size; i++) { int v = a[i]; for (j = i; j >= h && a[j - h] > v; j -= h) a[j] = a[j - h]; if (i != j) a[j] = v; } } while (h != 1);}

Sample Program

• Sample input/output– ./sample 11 14– Output: 0 11

• This is an obvious error but how do we find it?

• Use the TRAFFIC Principle

The TRAFFIC Principle

• Track• Reproduce• Automate• Find origins• Focus• Isolate• Correct

The TRAFFIC Principle

• Track– Tracking the problem is rather easy due to the size of the program– Just running the program you found the error.– In most cases searching for a bug is harder because most programs are much

larger

• Reproduce– Reproducing the error is rather easy. You just need to invoke the exe again.

• ./sample 22 33 Output: 0 22• ./sample 123 23 Output: 0 23• ./sample 123 23 11 15 Output: 0 11 15 23

– Once again in larger programs it can be much harder to reproduce the bug because most of the time it only happens under a small set of conditions

• Ex/ be using gcc 4.2.1, input1 = 23 and input2 must be larger than input1

The TRAFFIC Principle

• Automate– if the example program were a more complex program, we would

have to think about how to automate the failure (in that we want to reproduce the failure automatically) and how to simplify its input such that we obtain a minimal test case. (textbook)

The TRAFFIC Principle

• Find origins– In the example program we can step through the program(because it is small)– If we print the array that is sent to the shell_sort function we get for input 123

and 22 we get

– We can see that this is not the correct size, which causes a third element to be created, and that third element is 0.

The TRAFFIC Principle

• Focus– After seeing that there is one more element than there should be, we can focus

on it and investigate further.– We can see if the shell_sort is working properly by using the same print

statement at the end of the shell sort. We get the following output, which is now sorted.

– We can conclude that the sorting algorithm is correct but the size is causing the issue

The TRAFFIC Principle

• Isolate– We can conclude the most likely reasoning for the bug is that a[] is created with

one more element than it should be. – We can identify the original line of code that called shell sort

• When looking into the call shell_sort(a, argc); we can then realize that argc is the number of arguments put in by looking at the main input parameters.

• Note argc of ~$./a.out 4 5 is 3– 1) ./a.out– 2) 4– 3) 5

– We can now see the bug is caused by a confusion is in argc. We assume it would just contain the numbers and not the first argument “./a.out”

The TRAFFIC Principle

• Correct– Once we know the defect is caused by a misunderstanding of argc, we can fix

the problem by subtracting “1” from the argc. This is because we now know that we need to subtract the first argument to get the actual size of the array.

– So we can change the shell_sort call from shell_sort(a, argc); to shell_sort(a, argc-1);

The TRAFFIC Principle

• Correct program

From Defect to Failure

Efficiency of Testing

• How defects and failures are related– A defect is an error or a bug in the application which is created. A programmer

while designing and building the software can make mistakes or error. These mistakes or errors mean that there are flaws in the software.

– When the result of the software application or product does not meet with the end user expectations or the software requirements then it results into a bug or defect. These defects or bugs occur because of an error in logic or in coding which results into the failure or unpredicted or unanticipated results.

– In other words a defect can turn into a failure at runtime– (http://istqbexamcertification.com/what-is-defect-or-bugs-or-faults-in-software-testing/)

• Can we find all errors in the code?– NO. testing can only find errors and not the lack of errors(Dijkstra). – Some errors may not come from your program but libraries you use or even in

rare cases your complier

Search In Time And Space

• Search from defect to failure in time and space– You have an error but where did it come from– A failure could be caused by a chain of infections that could

potentially go through ever variable in the program.– So you could potentially need to search the entire program to

find the origin of the defect. – As you can imagine, for a very large project this can take a

massive amount of time Book image

Book image

Search In Time And Space

• Examples of the complexity of this search – Lets say we have a medium program 100,000+ line 2,500+ variables (a

variable every 40 lines)– Lets say that each variable is referenced 5 times during execution– Lets also say that we have an error somewhere in the program and the origin

is unknown– Using the debugger and stepping through the program it takes 5 seconds to

check each variable and reference for the defect– 2,500(var)*5(ref)*5(sec) = 250,000 sec / 60 sec = 4,166 min / 60 min = 69.4 hr– it could take 69 hours if you had to check every reference of every variable

Find Origins

• Deduce value origins– Example: the sample program(from earlier)

• Where did the 0 come from in the output• We had to check the function make sure it was not the cause and only then did we

find the origin of the 0.• We had to separate the relevant from the irrelevant values• Then finally trace from the effect to the cause.

• Separate relevant from irrelevant– A variable value is the result of a limited number of earlier variable values.

Thus, only some part of the earlier state may be relevant to the failure (textbook)

Search In Time

• Observe a transition from sane to infected– Sane state – no infection to propagate – In the example of the sample program we observed the transfer from a sane to

an infected state at the call to shell_sort. When the value of the size becomes incorrect. Once that value was infected it propagated all the way through the function and into the output.

Observing A Run

• Trace the execution for all variables for all steps of execution

• Here is an illustration for the sample program with inputs 14 11

• It is step through tracing each variable

Book image

Specific Observation

• We can observe at a specific location by using a debugger or print statements

• This is also the transition from sane to infected

State is infected at the call to shell_sort

Finding Cause

• The picture above illustrates the space of a program.• edges = references • Vertices = variables

See how vast it can be.

Book image

Finding Cause

• Search in space– When we search the space we look for variables and references – For example, the variable argc both the infected and sane state have this

variable – We can look though this to help us find the problem

http://www.whyprogramsfail.com/slides.php

Finding Cause

• Searching Time– When we search time we look at variables and references over

time– For example the argc variable

• In the failing run of the sample we would watch a specific variable in this case argc because we have found this to be the cause of the infection

http://www.whyprogramsfail.com/slides.php

As you can see in the figurewe track argc. Doing this we find that argc 3 is responsible for a[2]

Automate Debugging: Program Slice

• Program Slicing – Separating the part of a program or program run relevant to the

failure. In Figure below, we can see that only a fraction of the state actually could have caused the failure. Applied to sample program, a program slice could determine that a[0] got the zero value because of the values of a[2] and size, which is already a good hint. (textbook)

Book image

Automated Debugging: Observing State

• Stop the program to observe the entire state– This is done using a debugger or a print statements – Using a debugger on the sample program, we would be able to observe the

values of a[] and size at any moment in time without changing or recompiling the program. (textbook)

Stop and see what is happening

Book image

Automated Debugging: Watching State

• Watch a part of the state to find changes during execution– This allows us to find the precise moment during execution that the state

becomes infected– For example, in the sample problem using a debugger, we would be able to

watch the value of a[0] to catch the precise statement that assigns the zero from a[2].

Watch that specific variables tosee the point that it gets infected

Automated Debugging: Assertion

• Specify expected values and check for them– This is used at the beginning and end of function to make sure variables should

be what they should be during runtime.– If the assertion passes that means everything is good and continue – If the assertion fails this means that possibly a state has become infected.

Make sure that the variables are what they should be otherwise throw an error

Automated Debugging: Anomalies

• Observe the difference between passing and the failing runs– For example, in the sample problem we can compare the coverage of the two

runs sample 11 (passing) and sample 11 14 (failing). It turns out that the statements where a[j] is assigned a value are executed only in the failing run, but not in the passing run. Thus, if we are looking for a zero value in a[0] these two lines might be a good starting point. (textbook)

Look at variables on different runs to see if you can pin point the problem

Book image

Automated Debugging: Cause-Effect Chain

• Identify which variable caused the failure• applies delta debugging to program states, thus identifying in each

state which particular variable(s) caused the failure. This results in a cause–effect chain, as sketched below. Although causes are not necessarily errors, they help to narrow down the relevant elements of a failure. (textbook)

• Note: causes are not necessarily errors!

The illustration shows a chain that starts at the error origin and shows the propagation of the error through other variables

Book image

Automated Debugging: Simplified Inputs

• Simplify the input so that each part of the input contributes to the failure

• The chart is of an algorithm that narrows down the test case to the inputs that causes the failure.

• This then allows you to trace through your code to find out why a particular input combination causes a failure

• http://web.stanford.edu/class/cs295/papers/issta2000.pdf