Software Testing Strategies and Techniques

But First… Where have we been in this class

gathering requirements defining the work to be done estimating time and effort scheduling work analyzing risks measuring our work

Where are we going next testing our product managing change

Starter Questions What is the purpose of testing software?

Can we be sure the code is 100% correct?

How much "correctness" should we shoot for?

What needs to be tested?

How do we test those parts/aspects?

Given a finite schedule, what should a tester spend their time on?

Given a finite budget, what can a manager do to improve testing effectiveness?

Testing Terms

Verification - is the product correct

Validation - is it the correct product

Laws of TestingLaws of Testing

The best person to test your code is someone else.

A good test is one that finds an error.

Testing can not prove the absence of errors.

Complete test coverage is impossible, so concentrate on problem areas.

It cost a lot less to remove bugs early.

Testing Stages

Unit Testing modules of code

Integration Testing design

Validation Testing requirements

System Testing system engineering

Unit Testing Techniques

White Box Testing testing a module of code based on the source code example - basis path testing

Black Box Testing testing a module based just on what it is supposed

to do example - boundary value analysis

Code Coverage

Types of "Code Coverage" Statement coverage

each line of code has been tested

Conditional coverage each true/false decision has been tested

Path coverage each unique path through the code has been tested

Level of Coverage

Coverage Types are Apples and Oranges 100% statement coverage ≠ 100% path coverage

50% path coverage is probable better than 90% statement coverage

100% path coverage is usually impractical N if-statements create up to 2n paths

loops are worse

Basis Path TestingObjective is to test each conditional statement

as both true and false

1. Draw a Flow Graph

2. Determine the Cyclomatic Complexity count the regions, or CC = E - N + 2

3. Max Number of tests = CC

4. Derive a basis set of independent paths

5. Generate data to drive each path

Examplevoid delete (int val, struct node **head){ struct node *temp, *prev; prev = NULL; temp = *head;1 while (temp!=NULL)2 if (temp->value == val)3 break; else {4 prev = temp; temp = temp->next; }5 if (prev == NULL) {6 *head = (*head)->next; } else {7 prev->next = prev->next->next;

1 - while

5 - if 2 - if

6 7 3 - break 4

8

Example Continued

Complexity = 4

Basis Paths 1 2 4 1 5 7 8 1 2 3 5 7 8 1 2 3 5 6 8 1 5 6 8

1 - while

5 - if 2 - if

6 7 3 - break 4

8

Example Continuedvoid delete (int val, struct node **head){ struct node *temp, *prev; prev = NULL; temp = *head;1 while (temp!=NULL)2 if (temp->value == val)3 break; else {4 prev = temp; temp = temp->next; }5 if (prev == NULL) {6 *head = (*head)->next; } else {7 prev->next = prev->next->next;

Path 1 2 4 1 5 7 8 1 to 2 non empty 2 to 4 deleting non head 5 to 7 deleting non head

Path 1 2 3 5 7 8 1, 2, 3 deleting the head 5 to 7 deleting non head

Path 1 2 3 5 6 8 1, 2, 3 deleting the head 5 to 6 deleting the head

Path 1 5 6 8 1 to 5 empty list 5 to 6 deleting the head

Generating Paths

If a path is not possible, then re-generate the list of basis paths

An algorithm to generate only feasible paths is impossible (provably un-decidable)

Boundary Value Analysis

Errors usually happen at the ends, not in the middle

Example delete the head, size=1 delete the head, size>1 delete from end of the list, size>1 delete when size=0 delete when size≠0, not in list

Example

Create some test cases to verify the new "add alarm" feature.

Homework Results

1. 12:00am

2. 12:01am

3. 11:59am

4. 12:00pm

5. 12:01pm

6. 1:00pm

7. 12:59pm

12:00am 12:00pm 11:59pm

Mostcommon

error

Why?

11:59 = 11:59am12:00 = 12:00pm12:01 = 12:01pm13:00 = 1:00pm

Integration Testing

Bottom-Up requires lots of drivers gives real answers early big errors are not found until the end

Top-Down requires lots of stubs gives fake answers until the very end big errors found early

Sandwich Depth-First

A

DCB

GFE

Objective is to test the interfaces between modules

Integration Testingnecessitates

Regression Testing

When do we stop unit and integration testing?

Answer: when the cost of finding bugs becomes greater than the cost of removing bugs.

Time

numberof bugsfound

per day

Validation Testing

Alpha Testing

Beta Testing

Objective is to test the Conformity to Requirements

Usability Testing

Interface Design Heuristics

Cognitive Walkthroughs

System Testing

Recovery Testing

Security Testing

Stress Testing

Performance Testing

http://www.mscsoftware.com/success/images/ASL.jpg

Types of Testing Tools Regression Testing

retest to find side-effects Database Testing

test the queries build tests based on tables and fields load test the database

Load Testing simulate a lot of users - stress test

Functional Testing record a user's actions try lots of random user actions build test cases based on Use Case Scenarios

Testing Web Applications Unit Testing

Example - load testing

http://www.gamcom.com/testingdeliveryexample.htm

Example - functional testing - WinRunner

To create a test, Mercury WinRunner simply records a typical business process by emulating user actions, such as ordering an item or opening a vendor account. During recording, you can directly edit generated scripts to meet the most complex test requirements.

Next, testers can add checkpoints, which compare expected and actual outcomes from the test run. Mercury WinRunner offers a variety of checkpoints, including test, GUI, bitmap, and Web links. It can also verify database values to ensure transaction accuracy and database integrity, highlighting records that have been updated, modified, deleted, and inserted.

With a few mouse clicks, Mercury WinRunner’s DataDriver Wizard lets you convert a recorded business process into a data-driven test that reflects the unique, real-life actions of multiple users. For further test enhancement, the Function Generator presents a quick and reliable way to program tests, while the Virtual Object Wizard enables you to teach Mercury WinRunner to recognize, record, and replay any unknown or custom object.

As Mercury WinRunner executes tests, it operates the application automatically, as though a real user were performing each step in the business process. If test execution takes place after hours or in the absence of a QA engineer, Mercury WinRunner’s Recovery Manager and Exception Handling mechanism automatically troubleshoot unexpected events, errors, and application crashes to ensure smooth test completion.

Example - unit testing java code

Teams that have tried manual creation of unit tests, will find that the automation provided by Agitator makes developer testing practical for the first time. Organizations that have deployed automated solutions for requirements management, configurations management, systems, load, and performance testing will find that Agitator is the perfect complement to deliver improved software quality and reduced overall software lifecycle costs.

It is said that an effort is not worth doing if it can not be measured. Developer Testing is no different. For successful application of Developer Testing, a total of 20-40% of time dedicated to the project will be spent on the creation and execution of unit tests. It is therefore critical that teams can evaluate that they are allocating resources in the best possible way. The Agitar Management Dashboard is the first of its kind, created specifically to monitor and manage Developer Testing efforts.

Rest of Semester …Rest of Semester …

Nov 6 - Configuration Management Nov 13 - Software Quality Assurance Nov 20 - Quality Assurance Programs Nov 27 - Student Presentations