black box software testing - part 12

8/3/2019 Black Box Software Testing - Part 12

1/29

Black Box Software Testing Copyright 2003 Cem Kaner & James Bach 499

Black Box Software Testing

Part 12.

Introduction toTest Documentation

Co-authors: Cem Kaner & James Bach

AcknowledgementSome of this material is from the Third Los Altos Workshop on Software Testing (LAWST), February 7 and 8,1998. I founded the LAWST and was co-organizer of LAWST 3.

At LAWST, we discussed test documentation (test planning strategies and materials). The agenda item was:

How do we know what test cases we have? How do we know which areas of the program are well covered and which are not?

How do we develop this documentation EFFICIENTLY? As many of you know, I despise thick test plans and I begrudge everymillisecond that I spend on test case documentation. Unfortunately, some work is necessary. My question is, how little can we get awaywith, while still preserving the value of our asset?

The following people attended LAWST 3: Chris Agruss, James Bach, Karla Fisher, David Gelperin, KennethGroder, Elisabeth Hendrickson, Doug Hoffman, III, Bob Johnson, Cem Kaner, Brian Lawrence, Brian Marick,Thanga Meenakshi, Noel Nyman, Jeffery E. Payne, Bret Pettichord, Johanna Rothman, Jane Stepak, MeloraSvoboda, Jeremy White, and Rodney Wilson.


2/29


Copyright Notice

These slides are distributed under the Creative CommonsLicense.

In brief summary, you may make and distribute copies of these

slides so long as you give the original author credit and, if youalter, transform or build upon this work, you distribute theresulting work only under a license identical to this one.

For the rest of the details of the license, seehttp://creativecommons.org/licenses/by-sa/2.0/legalcode.


3/29


Definitions

The test planning documents include:

The Testing Project Plan, which identifies classes of tasks and

broadly allocates people and resources to them;

High-level designs for test cases (individual tests) and test

suites (collections of related tests);

Detailed lists or descriptions of test cases;

Anything else that you would put in a hard copy or virtual

binder that documents the collection of test cases that you will

develop and run.

This collection of materials is sometimes called the test planand sometimes called the test documentation.


4/29


Levels of Analysis

1-Variable

Basic user interface verification

Filters

Many testers stop here, at the most superficial level of testing

2-Variable

Shared or related filters

Logical relationships among variables

Multi-Variable

The core problem is the vastness of the space of test cases. We

need a sampling strategy.


5/29


Basic Test Documentation Components

Lists: Such as lists of fields, error messages, DLLs

Outlines: An outline organizes information into a hierarchy oflists and sublists

Such as the testing objectives list later in the course notes

Tables: A table organizes information in two dimensionsshowing relationships between variables.

Such as boundary tables, decision tables, combination test tables

Matrices: A matrix is a special type of table used for datacollection.

Such as the numeric input field matrix, configuration matrices

Refer to Testing Computer Software, pages 217-241. For more

examples, see page Testing Computer Software, page 218.


6/29


Configuration Planning Table

Var 1 Var 2 Var 3 Var 4 Var 5Config 1 V1-1 V2-1 V3-1 V4-1 V5-1

Config 2 V1-2 V2-2 V3-2 V4-2 V5-2

Config 3 V1-3 V2-3 V3-3 V4-3 V5-3

Config 4 V1-4 V2-4 V3-4 V4-4 V5-4

Config 5 V1-5 V2-5 V3-5 V4-5 V5-5

Config 6 V1-6 V2-6 V3-6 V4-6 V5-6

This table defines 6 standard configurations for testing. In later tests, the lab

will set up a Config-1 system, a Config-2 system, etc., and will do its

compatibility testing on these systems. The variables might be software orhardware choices. For example, Var 1 could be the operating system (V1-1 is

Win 2000, V1-2 is Win ME, etc.) Var 2 could be how much RAM on the

computer under test (V2-1 is 128 meg, V2-2 is 32 meg., etc.). Var 3 could be

the type of printer, Var 4 the machines language setting (French, German,

etc.). Config planning tables are often filled in using the All Pairs algorithm.


7/29Black Box Software Testing Copyright 2003 Cem Kaner & James Bach 505

Configuration Test Matrix

Config 1 Config 2 Config 3 Config 4 Config 5 Config 6Test 1 Pass Pass Pass Pass Pass

Test 2 Fail Pass Pass Pass

Test 3 Pass Pass Pass Pass Pass

Test 4 Pass Fail Fail Pass

Test 5 Fail Pass Fail Pass Pass

This matrix records the results of a series of tests against the 6 standard

configurations that we defined in the Configuration Planning Table.

In this table, Config 1 has passed 3 tests, failed 1, and hasnt yet been tested

with Test 2. Config 6 is still untested.




Part 13.

Scripting:

An Industry Worst Practice



Scripting

COMPLETE SCRIPTING is favored by people who believe thatrepeatability is everything and who believe that with repeatablescripts, we can delegate to cheap labor.

1 ____ Pull down the Task menu

2 ____ Select First Number

3 ____ Enter 3

4 ____ Enter 2

5 ____ Press return

6 ____ The program displays 5


10/29


Scripting: Another script format

Step

#

Check

?

What to

do

What to

see

Design

notes

Observation

notes

1. ____ Pull down

task menu

Task menu

down

This starts the

blah blah test,

with the blah

blah goal


11/29


Scripting: The Bus of Testing

Scripting is the Greyhound Bus of software testing:

Just relax and leave the thinking to us.

To the novice, the test script is the whole tour. The tester goes

through the script, start to finish, and thinks hes seen whatthere is to see.

To the experienced tester, the test script is a tour bus. When

she sees something interesting, she stops the bus and takes a

closer look.

One problem with a bus trip. Its often pretty boring, and you

might spend a lot of time sleeping.


12/29


Scripting (Manual Test Cases)

Worst of all worlds:

Not automated

Not maintainable

No communication of analysis

NOT REPEATABLE!


13/29


Scripting

Should we create scripts as pseudocode, inpreparation for test automation?

Suggestion made at STAR 97

But most (88%) of bugs were found during

creation of the scripts (my experience too).

Maybe the benefit lies in

structured exploration,rather than regression.


14/29


Never Do ANY kind of Scripting?

Checklists (as distinct from scripts) have their place. Forexample, think of releasing a product: Many very different tasks

All of them must be done

The task is rarely done, so many steps may be forgotten.

I prefer to tell testers what to test (what issues to cover), nothow to do the tests. Teaching people how is a matter oftraining, not something that I record time and time again in thetest plan. A checklist will sometimes be the right way to presentthe list of issues.

I think the level of detail is sufficient if I can successfully passthe section to a reasonably experienced tester who is a littlefamiliar with the program and be confident that she can figureout what the test cases are and how to run them.


15/29



Part 14.

Requirements Analysis forTest Documentation


16/29


Problems with the (allegedly) standard

approach

It is essential to understand your requirements fortest documentation.

Unless following a standard helps you meet yourrequirements, it is empty at best, anti-productive atworst.


17/29


Problems with the (allegedly) standard approach

IEEE Standard 829 for Software Test Documentation Test plan

Test-design specification

Test-case specification

Test-case specification identifier

Test items Input specifications

Output specifications

Environmental needs

Special procedural requirements

Intercase dependencies

Test-procedure specification

Test-item transmittal report

Test-log

We often see

one or morepages per

test case.


18/29



approach

What is the documentation cost per test case?

What is the maintenance cost of the documentation, per test case?

If software design changes create documentation maintenance

costs, how much inertia do we build into our system? How muchdoes extensive test documentation add to the cost of late

improvement of the software? How much should we add?

What inertia is created in favor of invariant regression testing?

Is this incompatible with exploratory testing? Do we always want

to discourage exploration?


19/29



approach

What is the impact on high-volume test automation?

How often do project teams start to follow 829 but then give it up

mid-project? What does this do to the net quality of the test

documentation and test planning effort?

WHAT REQUIREMENTS DOES A STANDARD LIKE THIS

FULFILL?

WHICH STAKEHOLDERS GAIN A NET BENEFIT FROM

IEEE STANDARD DOCUMENTATION?

WHAT BENEFITS DO THEY GAIN, AND WHY ARE THOSEBENEFITS IMPORTANT TO THEM?


20/29


Standards:

ISO 9000-3 The basic thinking underlying ISO 9000 goes like this:

Decide what youre going to do (create and document a process).

Do what you said youd do.

Document what youve done in a way that lets an auditor checkwhat youve done against your process.

Risks It can be difficult to build in room for exploratory testing. To

some degree (perhaps a large degree), this depends on the auditor.

The style and extent of documentation needed to clue in a thirdparty (auditor) might go beyond your internal needs. On finite

budgets, where do we draw the line? What is the best tradeoff between process development and test

execution? This must be a practical question, not a religiousquestion.


21/29


Requirements

There are many different notions of what a good set of test

documentation would include. Before spending a substantial

amount of time and resources, its worth asking what

documentation should be developed (and why?)

Test documentation is expensive and it takes a long time toproduce. If you figure out some of your main requirements first,

you might be able to do your work in a way that achieves them.


22/29


Defining documentation requirements

Stakeholders, interests, actions, objects

Who would use or be affected by test documentation?

What interests of theirs does documentation serve or disserve?

What will they do with the documentation?

What types of documents are of high or low value?

Asking questions

Context-free questions

Context-free questions specific to test planning

Evaluating a plan


23/29


Discovering Requirements

Requirements

Anything that drives or constrains design

Stakeholders

Favored, disfavored, and neutral stakeholders

Stakeholders interests

Favored, disfavored, and neutral interests

Actions

Actions support or interfere with interests

Objects


24/29


What is your groups mission?

Find important problems

Assess quality

Certify to standard

Fulfill process mandates

Satisfy stakeholders

Assure accountability

Advise about QA

Advise about testing

Advise about quality

Maximize efficiency

Minimize time

Minimize cost

The quality of testing depends on which of these

possible missions matter and how they relate.

Many debates about the goodness of testing

are really debates over missions and givens.


25/29


Test Docs Requirements Questions

Is test documentation a product or tool?

Is software quality driven by legal issues or by marketforces?

How quickly is the design changing?

How quickly does the specification change to reflectdesign change?

Is testing approach oriented toward proving conformanceto specs or nonconformance with customer expectations?

Does your testing style rely more on already-defined testsor on exploration?

Should test docs focus on what to test (objectives) or onhow to test for it (procedures)?

Should the docs ever control the testing project?


26/29



If the docs control parts of the testing project, should thatcontrol come early or late in the project?

Who are the primary readers of these test documents and howimportant are they?

How much traceability do you need? What docs are you tracing

back to and who controls them?

To what extent should test docs support tracking and reportingof project status and testing progress?

How well should docs support delegation of work to newtesters?

What are your assumptions about the skills and knowledge ofnew testers?

Is test doc set a process model, a product model, or a defectfinder?


27/29



A test suite should provide prevention, detection, andprediction. Which is the most important for this project?

How maintainable are the test docs (and their testcases)? And, how well do they ensure that test changeswill follow code changes?

Will the test docs help us identify (and revise/restructurein face of) a permanent shift in the risk profile of theprogram?

Are (should) docs (be) automatically created as a

byproduct of the test automation code?


28/29


Example: Early vs. Late Planning

Benefits of early planning

Scheduling and staffing (unless things change)

Early opportunity to review (if people can understand your work)

Costs of early planning

Any delays in finding bugs are expensive. You should startfindingbugs ASAP.

Early investment in test case design is risky if the product design issubject to change.

You will keep learning over time. Planning early sometimesprecludes groups from inventing new tests later.

Depth of testing should reflect risks, many of which are unclear untilmid-project.

MY USUAL GOAL: Do just-in-time test planning without losing keybenefits of early work.


29/29

Black Box Software Testing Copyright 2003527

Ultimately, write a mission statement

Try to describe your core documentation requirements in onesentence that doesnt have more than three components.

Examples:

The test documentation set will primarily support our efforts to

find bugs in this version, to delegate work, and to track status.

The test documentation set will support ongoing product and testmaintenance over at least 10 years, will provide training material

for new group members, and will create archives suitable for

regulatory or litigation use.

black box software testing - part 12

Documents