CIS-74 Computer Software Quality Assurance

Systematic Software Testing

Chapter 5:

Analysis and Design

Creating an Inventory

1. Gather reference materials

2. Form a brainstorming team

3. Determine test objectives

4. Prioritize objectives

<continued on next slide>

Creating an Inventory

5. Parse (organize) objectives into lists.

6. Create an inventory tracking matrix

– Left-hand column of objectives should be in prioritized order from step 4.

– This step known as requirements traceability.

<continued on next slide>

Creating an Inventory

7. Identify tests for unaddressed conditions, adding/modifying test cases as needed.

8. Evaluate each inventory item for adequacy. (And add/modify test cases as needed.)

9. Maintain the testing matrix.

Black-Box vs. White-Box

Black-box testing - testing done without knowledge of the internal workings of the program/system being tested.

White-box testing - testing done with knowledge of the internal workings of the program/system being tested.

White-box testing is also called structural testing because it’s based on structure of the code.

Black-Box Science:

Equivalence Partitioning

An equivalence class is formed by a group of tests which:

• All test the same thing.

• If one catches a bug, the others do too.

• If one doesn’t catch a bug, the others won’t either.

Black-Box Science:

Boundary Value Analysis

Test for:

• Exact boundaries

• Value immediately above upper boundary

• Value immediately below lower boundary

Black-Box Science: State-Transition Diagrams

A state machine is a thing which depends not just on current input, but also on past input, for determining output.

A transition occurs when a state is changed by its input from one state into another.

To use in black-box testing: Create a state-transition diagram.

Create a test case for every arrow (transition) from the diagram.

Black-Box Science: Orthogonal Arrays

An Lx(Yz) orthogonal array is a two-dimensional array with:

• X rows

• Z columns

• 1..y in cells of matrix

Black-Box Science: Orthogonal Arrays

• Useful with compatibility testing in particular

• What text did not say:

• How to create an orthogonal array

• How to use an orthogonal array when the variables do not all have the same number of values (as did the example in Table 5-9)

Black-Box Art: Ad Hoc Testing

• Often based on systematic techniques

• Test cases should be created for all bugs found

• Can locate an entire class of tests missed using more formal means

• Fun!

Black-Box Art: Random Testing

• All data fields generated randomly, often with a tool

• Can lead to too many negative tests - those with an invalid input, and expected results of an error message

• Not very useful

• At best, useful for “crash-proofing”

Black-Box Art: Semi-Random Testing

• Random combinations of legitimate values for variables

• Gives a little added confidence to systematic testing

Black-Box Art: Exploratory Testing

• Test design and execution occur concurrently

• Test results may prompt user to delve deeper into that area

• Neither ad hoc nor random in nature

• Productive areas of test are immediately expanded

• Viewed as complementary to structured testing by authors

White-Box Science: White-Box Testing

• “White-box” should be “clear-box” or “glass-box” in authors’ views. (Why?)

• Normally viewed as a technique for unit testing, but can also be used with integration & system testing

• Primarily about path testing (paths through the code)

White-Box Science: Coverage Techniques

• Statement coverage - # of lines of code executed by the tests

• Decision or branch coverage - # of decisions executed by the tests

• Path coverage - # of paths executed by the tests

Test Design Documentation:IEEE Test Design Specification

• One TDS per feature from the “Features to be Tested” from the corresponding test plan

• Test identification section includes test case id’s and high-level description for each test case but not info on how to execute them.

Test Design Documentation:IEEE Test Design Specification

• Test cases may not be sequential because TDS covers all tests necessary to test one feature--some tests for other features may be reusable.

• Feature pass/fail criteria establish success or failure for the TDS’s feature--not individual test cases.

Test Design Documentation:Test Case Specification

Two approaches:

• IEEE Template for Test Case Specification• Focuses on the “what”

• Spreadsheet • Very common method• Can also be used to record results

Test Design Documentation:IEEE Test Procedure Specification

Focuses on “how” tests will be run

Can be run via scripts--automated test procedures. These “scripts” are normally written in a scripting language such as Perl or a UNIX/Linux shell language.

“Scripts” also refer to testing instructions for manually-executed tests.

Review Questions

Which test design technique was covered in depth by the first part of Chapter 5?

Which test design technique was covered in depth by the first part of Chapter 5?

Inventories

What are the column headers in an inventory tracking matrix?

What are the column headers in an inventory tracking matrix?

Test case identifiers.

How are the objectives/inventory items in the column of an inventory tracking matrix ordered?

How are the objectives/inventory items in the column of an inventory tracking matrix ordered?

Priority order (highest to lowest)

With respect to an inventory tracking matrix, what is the name for the process of adding “X”s to those non-header cells which represent test case coverage of an objective/inventory-item?

With respect to an inventory tracking matrix, what is the name for the process of adding “X”s to those non-header cells which represent test case coverage of an objective/inventory-item?

Requirements traceability

What is the name for testing based upon the requirements, i.e., the internal workings of the system under test are unknown?

What is the name for testing based upon the requirements, i.e., the internal workings of the system under test are unknown?

Black-box testing

What is another name for black-box testing?

What is another name for black-box testing?

Behavioral testing.

What is white-box testing?

What is white-box testing?

Testing in which the internal workings of the system under test are known to the tester.

What is another name for white-box testing?

What is another name for white-box testing?

Structural testing. (Why?)

Glass-box testing

Clear-box testing

Translucent-box testing

What is coincidental correctness?

What is coincidental correctness?

Right results for wrong reason. (A test case passes even though the input was processed incorrectly.)

What are the three defining aspects of a group of tests making up an equivalence class?

What are the three defining aspects of a group of tests making up an equivalence class?

1. They all test the same thing

2. If one catches a bug, the others will too.

3. If one fails to catch a bug, the others will also fail.

What are the three areas to test when using boundary analysis?

What are the three areas to test when using boundary analysis?

1. The exact boundaries

2. The value immediately above the upper boundary

3. The value immediately below the lower boundary

Which black-box science technique involves creating a table which lists all possible inputs and outputs?

Which black-box science technique involves creating a table which lists all possible inputs and outputs?

Decision tables

In a state-transition diagram, how are the states and transitions depicted?

In a state-transition diagram, how are the states and transitions depicted?

State - circle with label inside

Transition - directed arc between two states

What is an orthogonal array?

What is an orthogonal array?

A two-dimensional array of integers

Any two columns contain all combinations of the integers contained in the array

What is meant by L9(34)?

What is meant by L9(34)?

An orthogonal array with 9 rows, 4 columns, and cell contents ranging from 1 through 3, inclusively.

What is an example of a pair-wise defect?

What is an example of a pair-wise defect?

A web site that does not work properly with a particular browser and OS even though it works on many other browser/OS combinations.

Example: Bug 7019

What kind of testing involves generating miscellaneous data, often using a tool?

What kind of testing involves generating miscellaneous data, often using a tool?

Random testing

What is the name for a test in which the tester supplies an invalid input in the expectation of receiving an error message?

What is the name for a test in which the tester supplies an invalid input in the expectation of receiving an error message?

Negative test

What is the name for testing in which the test design and test execution are conducted concurrently?

What is the name for testing in which the test design and test execution are conducted concurrently?

Exploratory testing

How do the authors view exploratory testing vs. structured testing?

How do the authors view exploratory testing vs. structured testing?

As a complementary process

What names do the authors claim are better than “white-box testing?”

What names do the authors claim are better than “white-box testing?”

Clear-box testing or glass-box testing.

What metric describes the complexity of a software module?

What metric describes the complexity of a software module?

Cyclomatic Complexity

End of Chapter 5