test script standards - purdue university - department …® 5300 software july 13, 1994 test script...

1999, Tellabs Operations, Inc., 4951 Indiana Ave., Lisle, IL 60532 (708) 969-8800

Test Script Standards

ID# 5300000167, Revision 2.0, July 13, 1994

TITAN ® 5300 Software

TITAN® 5300 Software July 13, 1994Test Script Standards Revision 2.0

Page i

TELLABS CONFIDENTIAL

Table of Contents

1.0 Introduction..............................................................................................................11.1 Purpose, Scope, and Field of Application ........................................................................... 1

1.1.1 Purpose.................................................................................................................. 11.1.2 Scope.....................................................................................................................11.1.3 Field of Application .............................................................................................. 1

1.2 Revision History.................................................................................................................. 1

1.3 Reference............................................................................................................................. 1

1.4 Definitions ........................................................................................................................... 2

1.5 Acronyms ............................................................................................................................ 2

2.0 Unit Test Scripts: White-Box Testing ......................................................................32.1 Unit Test Design.................................................................................................................. 3

2.2 The Standard ....................................................................................................................... 3

2.3 Generating Test Cases ......................................................................................................... 5

2.4 An Example of Generating a Unit Test Script..................................................................... 52.4.1 The Structure Chart and Code............................................................................... 52.4.2 The Unit Test Script.............................................................................................. 7

3.0 Software Interface Test Scripts: Gray-Box Testing .................................................93.1 Software Interface Test Design ........................................................................................... 9

3.2 The Standard ..................................................................................................................... 10

3.3 The Interface Test Inputs and Outputs for USadd() .......................................................... 12

3.4 The Interface Test Script for USadd() ............................................................................... 13

4.0 SWIT Scripts: Black-Box Testing .........................................................................154.1 SWIT Script Design ..........................................................................................................15

4.2 Determining Test Cases..................................................................................................... 16

4.3 The Standard ..................................................................................................................... 16

4.4 The SWIT Script for user_add on the TITAN 532L ......................................................... 18

4.5 Manpage for UTL_ADD_USER....................................................................................... 18

4.6 Inputs and Outputs for UTL_ADD_USER ....................................................................... 19

4.7 The Swit Script of UTL_ADD_USER.............................................................................. 20


, andto be

ribed.

iform

, lowlevelsoft-

1.0 Introduction

This document describes the TITAN® 5300 standards for writing unit test, interface testSWIT (software integration test) scripts. Examples are included, but the reader is assumedfamiliar with the Software Development Lifecycle Procedure[1]. Most of the processes deschere come fromThe Art of Software Testing by Glenford J. Myers, John Wiley & Sons, 1979[2]

1.1 Purpose, Scope, and Field of Application

1.1.1 Purpose

This document was created to assist developers in developing test scripts that follow a unstandard.

1.1.2 Scope

This document in a standard that must be followed in developing test scripts for high levellevel, and code designs. SWIT test scripts verify high level design, IT test scripts verify lowdesign, and unit test scripts verify code. For more information on how testing relates to theware lifecycle, please refer to the TITAN 5300 Software Verification and Validation Plan[3].

1.1.3 Field of Application

This document was created for the TITAN 5300 software developers.

1.2 Revision History

Rev Date Author Revision Description

1.0 2/90 Harpster Released for the TITAN 532L staff

2.0 6/94 Luka Updated information for TITAN 5300 Staff

1.3 Reference

[1] Software Development Lifecycle Procedure

[2] Glenford J. Myers,The Art of Software Testing, John Wiley and Sons, 1979.

[3] TITAN® 5300 Software Verification and Validation Plan

[4] TITAN® 5300 Unit Test Guidelines

[5] TITAN® 5300 SWIT Test Guidelines

Introduction Page 1



lence

ly have

tines0andd,

ctc-

inceuc-

rac-is

ts ofture

s of am iser-of

1.4 Definitions

boundary analysis Values just inside and outside the border between two equivaclasses

equivalence classes Breaking the parameter values into ranges that should essentialthe same effect when the function is called.

foreign subroutine A routine that is external to the design under test. Foreign subrouincluded library functions, operating system routines, TITAN 550code, and other purchased software. The difference between a stuba foreign subroutine is that a foreign subroutine is fully implementedesigned, verified, and can return multiple control values.

mspec In 532L HLD, this provides in textual form a description of the objeaction. In 5300 LLD, this describes in detail the functions in the struture chart.

stub A routine that is external to the design that is currently under test. Sthis routine is external, it is usually not implemented and returns a scessful control value.

white box testing This testing approach involves the examination of the internal chateristics of a subroutine/function. On the TITAN 5300 products, thtests the individual subroutines or procedures in a structure chart.

gray box testing This type of testing is concerned with examining the inputs/outputhe subroutines within a program. This testing exercises a strucchart or an object action.

black box testing In this approach, the test focuses on the input/output parameterprogram; therefore, the internal behavior and structure of the progranot the primary focus. On the TITAN 5300 products, these tests excise the inputs/outputs to an object relationship chart or a groupstructure charts.

1.5 AcronymsHLD High Level Design

LLD Low Level Design

OOD Object Oriented Design

ORC Object Relationship chart

IT Interface Test (gray box)

SWIT Software Integration Test (black box

UT Unit Testing)

Introduction Page 2



chart.

ini-

t in a

o be

ate-every

mod-

ment

g to the

enda-

2.0 Unit Test Scripts: White-Box Testing

2.1 Unit Test Design

Unit testing is a process of testing the individual subroutines or procedures in a structureThe motivations for unit testing are threefold:

• Unit testing is a way of managing the combinatorics of testing, since attention is focusedtially on smaller units of the application.

• Unit testing eases the task of debugging, since, when an error is found, it is known to exisparticular module.

• Unit testing introduces parallelism into the testing process by allowing multiple modules ttested simultaneously.

Unit testing is largely white-box oriented, which is concerned with testing the individual stments within a particular routine/procedure. The ultimate white-box test is the execution ofpath in the module, unfortunately this is usually infeasible.

2.2 The Standard

Unit test scripts must adhere to the following rules:

• There must be a unit test script for each subroutine, except for Foreign Subroutines. Eachule that is new for a particular structure chart, i.e., has an mspec, needs a unit test.

• All statements in the procedure/function must be executed at lease once

• Stubs can be used.

• The execution of the tests can be performed on the target system or a simulated environ(such a CodeCenter®)

• After each input, the output (if any) must be checked. The word “output” here means writinglobal variables, the return value of a subroutine, writing to output parameters, writing todatabase, and input parameters for subordinate routines.

• The unit test script must have the same name as the source file with “.ut” at the end.

• The following template must be used for unit test scripts with requirements and recommtions listed

Requirements:

• Header Section

• Test Number

• Summary Section

• Input and Output Section

•Status after each test case, which can be either Pass or Fail

Unit Test Scripts: White-Box Testing Page 3



• Date of approval

• Signature of Tester at end of testing.

/******************* BEGIN **************************

* COPYRIGHT DATE: mm/dd/yy BY TELLABS OPERATIONS, INC.

* ALL RIGHTS RESERVED

* NO PART OF THIS DOCUMENT MAY BE REPRODUCED WITHOUT

* THE WRITTEN PERMISSION OF TELLABS, INC.

* 4951 INDIANA AVENUE, LISLE, IL. 60532

*******************************************************

*

* Name:<Function name>

* Type:Unit Test Script

* Originator:<Author’s Name>

* File:<filename>

* Version:<version number>

* Reference:<LLD Reference>

* Description: <One line description of function>.

*

*

************************ END **************************

*/

Test #

Summary:.

<A description of the test case/condition being executed>

Input:

<Any inputs required to verify the summary/case condition>

Expected Output:

<Any return conditions, functions being executed with param-eters, database or global data updates>

Status:

<Pass or Fail>

•

•

•

Date:

Signature:




goal

lines

ownrary

2.3 Generating Test Cases

Finding the highest probability of finding bugs without generating too many test cases is theof unit testing. The methods of accomplishing this task are listed below:

• Finding Equivalence Classes

• Boundary-value Analysis

• Error Guessing

For more information on these methods, please refer to the TITAN 5300 Unit Test Guidedocument[4].

2.4 An Example of Generating a Unit Test Script

In this example, we’ll examine the structure chart shown in Figure 1. The only new routine shon this chart isUSadd() . The others don’t have mspecs on this chart since they are either libroutines or exist on their own structure charts. Therefore, onlyUSadd() needs a unit test to com-plete testing of this structure chart.

2.4.1 The Structure Chart and Code

FIGURE 1. Structure Chart for USadd()

The code forUSadd() is given below:

ERROR

USadd(name, level)

char *name;

SECURITY_LEVEL level;

{

register int uid, /* user id */

empty_uid = -1; /* first empty slot in list */

USadd

name,level

error

USencrypt

namepassword

DBread DBwrite

DB_EXCLUSIVE

tuple

ok

tupleok

strcmp

name,uidname

diff

strcpy

stra,strb




USTUPLE userlist[USMAXUSERS];/* list of users */

char password[USPWLEN]; /* default password */

ERROR err; /* return value */

ERROR DBread(),

DBwrite();

char *strncpy();

void USencrypt();

/*

* Get the user list.

*/

if ((err = DBread(DB_EXCLUSIVE,(genericptr) userlist)) != OK)

{

return err;

}

/*

* Search for an empty uid.

*/

for (uid = 0; uid < USMAXUSERS; uid++)

{

if (userlist[uid].us_name[0] == ‘\0’)

{

if (empty_uid == -1)

{

empty_uid = uid; /* found empty slot */

}

}

else if (strcmp(userlist[uid].us_name, name) == 0)

{

return E_US_DUPLICATE; /* duplicate name */

}

}

if (empty_uid == -1) /* no empty slots */

{

return E_US_TOOMANY;




}

/*

* We now have an empty spot to place the user in. * Copy in the info.

*/

(void)strncpy(userlist[empty_uid].us_name,name,USNAMELEN);

userlist[empty_uid].us_name[USNAMELEN] = ‘\0’;

userlist[empty_uid].us_level = level;

/*

* Give the person a default password. It’s the same as * their user name.

*/

USencrypt(name, password);

(void)strncpy(userlist[empty_uid].us_password,password, USPWLEN);

userlist[empty_uid].us_password[USPWLEN] = ‘\0’;

/*

* Save the changes to the list.

*/

return DBwrite((genericptr) userlist);

}

2.4.2 The Unit Test Script

The following test script shows some of the test cases for utl::add, user.

/******************* BEGIN **************************

* COPYRIGHT DATE: 01/01/90 BY TELLABS OPERATIONS, INC.





*******************************************************




*

* Name:Add user

* Type:Unit Test Script

* Originator:Stephen Harpster

* File: USadd.unit

* Version:1.1

* Reference: USadd.sce

* Description: Add a new user to the system.

*

*

************************ END **************************

*/

/*

* The following test could be considered error guessing orcommon

* sense

*/

Test Case 1:

Summary:

Adding a Successful User

Inputs:

name = “steve”

level = SL_CRAFT

userlist = (empty list)

Expected Output:

strncpy() called with new user name.

USencrypt() called with new password.

strncpy() called with encrypted password.

DBwrite() called with new user name and encrypted pass-word added to tuple.

USadd() returns OK.

Status: Pass

/*

* The following tests will verify boundary-value analysis

*/

Test Case 2.

Summary:

Attempt to add maximum users

Inputs:




rcisesx andtityuts toterface

name = “steve”

level = SL_CRAFT

userlist is USMAXUSERS -1

Expected Output:

uid is one less than USMAXUSERS

strncpy() called with new user name.

USencrypt() called with new password.

strncpy() called with encrypted password.

DBwrite() called with new user name and encrypted pass-word added to tuple.

USadd() returns OK.

Status: Pass

Test Case 3.

Summary:

Attempt to add to many users

Inputs:

name = “steve”

level = SL_CRAFT

userlis t is USMAXUSERS

Expected Output:

uid equals USMAXUSERS

USadd() returns E_US_TOOMANY.

Status: Pass

•

•

•

Date: 1/1/90

Signature: S Harpster

3.0 Software Interface Test Scripts: Gray-Box Testing

3.1 Software Interface Test Design

Whereas unit testing exercised individual modules in a structure chart, interface testing exea structure chart, or group of structure charts, as a whole. The process is both black-bowhite-box in nature (hencegray-box), that is, a structure chart is viewed as a single atomic enwith inputs and outputs. By giving the structure chart certain inputs, we expect certain outpbe generated. Determining what those inputs and outputs should be is the purpose of the intest script.

Software Interface Test Scripts: Gray-Box Testing Page 9



dia-es a

riables.rnd

reure

sses.

viron-

end.

om-

Inputs and outputs to a structure chart generally flow in and out of the top-most box on thegram. For example,USadd() in Figure 1 is fed a user name and security level and generatcompletion code.

Input and output can come from other areas too, however, such as the database or global vaIn Figure 1 again, some input is obtained throughDBread() . Under certain conditions, the usetuple is updated by callingDBwrite() . The interface test script must check these inputs aoutputs too.

3.2 The Standard

Interface test scripts must adhere to the following rules:

• On the TITAN 532L, there must be a interface test script for each object action or structuchart. For non-OOD 5300 products, there must be an interface test script for each structchart.

• Each interface test must provide full coverage of both the valid and invalid equivalence claThis can be shown by a list of all the inputs and outputs.

• Stubs can be used; however, the target hardware should be used when possible.

• The execution of the test script can be performed on the target system or a simulated enment (such a code center).

• The interface test script must have the same name as the structure chart with “.it” at the

• The following template must be used for interface test scripts with requirements and recmendations listed

Requirements:

• Header Section

• Test Case Number

• Summary Section



• Date of approval at end of testing

• Signature of Tester

Recommendations:

• All inputs and outputs listed at the beginning of the test script

/******************* BEGIN **************************









*******************************************************

*

* Name:<Name of Test>

* Type:Interface Test Script

* Originator:<Author>

* File:<filename>


* Reference: <Low level design reference>

* Description: <Description of Test Script>.

*

*

************************ END **************************

*/

Inputs: <List of Inputs required for Test Script>

Outputs:<List of Outputs for Test Script>

Test 1.

Summary:

<Description of Test Step>

Input:

<Inputs Required for Test>

Expected Output:

<Outputs from Test>

Status:

<Pass or Fail>

•

•

•

Date:

Signature:




base,

nce

withiner,

bound-es for

valid-

umontain

n on

-

3.3 The Interface Test Inputs and Outputs for USadd()

Looking back the example in Section 2, we can identify three inputs:name, level, anduserlist . “Inputs” can be reading global variables, input parameters, reading the dataoutput parameters from subordinate routines, or return values from a subordinate routines.

For each input, two groups are created called thevalid equivalence classand theinvalid equiva-lence class.The valid equivalence class lists valid values for each input; the invalid equivaleclass lists invalid values for each input. The two classes for our example are shown below:

Input Valid Invalid

name <= USNAMELEN > USNAMELEN

level SL_CRAFT 6

userlist not full full

DBread() name not in userlist name already in userlist

Inputs that are checked by the compiler, e.g., enumerated types in C, need not be includedan equivalence class. In this example,level is not an enumerated type, but an unsigned integhence the test.

When choosing test cases for the classes, experience has shown that cases which exploreary conditions have a higher payoff in detecting errors than cases that do not. Guidelinboundary conditions are as follows:

• If a condition specifies a range of values, write test cases for the ends of the range, and ininput/output test cases for situations just beyond the ends. For example, ifnum must be be-tween 1 and 6, then test fornum == 0, num == 1, num == 6, andnum == 7 .

• If a condition specifies a number of values, write test cases for the minimum and maximnumber of values and one beneath and beyond these values. For example, if a file can c1-255 records, test for 0, 1, 255, and 256 records.

• If the input or output is an ordered set, e.g., a sequential file, linear list, table, focus attentiothe first and last elements of the set.

You should note that invalid classes may be disjoint. Ifnumhas a valid range of 2 to 5, then the invalid class consists of those numbers less than 2 and greater than 5.

Enumerating the inputs gives the following list.

1. name < USNAMELEN

2. name == USNAMELEN

3. name > USNAMELEN

4. level == SL_CRAFT

5. level == 6

6. userlist not full




am-

7. userlist full

8. DBread() returns a list that does not containname

9. DBread() returns a list that containsname

Now we list all possible outputs.

a error = OK

b. error = E_US_DUPLICATE

c. error = E_US_TOOMANY

d. DBwrite() is called withname andlevel in userlist

Like unit test scripts, “outputs” include writing to global variables, the return value, output pareters, writing to the database, and input parameters to subordinate routines.

3.4 The Interface Test Script for USadd()

/******************* BEGIN **************************






*******************************************************

*

* Name:Add user

* Type:Interface Test Script


* File:USadd.sit

* Version:1.1

* Reference: USadd.sce


*

*

************************ END **************************

*/

Inputs:

1. name < USNAMELEN

2. name == USNAMELEN

3. name > USNAMELEN




4. level == SL_CRAFT

5. level == 6


7. userlist full

8. DBread() returns a list that does not contain name

9. DBread() returns a list that contains name

Outputs:

a. error = OK

b. error = E_US_DUPLICATE

c. error = E_US_TOOMANY

d. DBwrite() is called with name and level in userlist

Test 1.

Summary:

This test will verify that a user can be added to an

empty list

Input:

name = “steve”

level = SL_CRAFT


Expected Output:

error = OK

Status: Pass

Test 2.

Summary:

This test will verify that the maximum name can be added

Input:

name = “ANameWithTwentyChars”

level = SL_CRAFT

userlist = “steve”, SL_CRAFT

Expected Output:

error == OK

Status: Pass

Test 3.

Summary:




cep-

struc-

ch as

This test will verify that extremely long name is trun-cated

Input:

name = “AnExtremelyLongNameThatExceeds USNAMELEN”

level = SL_CRAFT


“ANameWithTwentyChars”, SL_CRAFT

Expected Output:

error = OK

Status: Pass

•

•

•

Test 8.

Summary:

This test will verify that duplicate names are handled

correctly.

Input:

name = “steve”

level = SL_CRAFT


Expected Output:

error = E_US_DUPLICATE

Status: Pass

Date:1/1/90

Signature: S. Harpster

4.0 SWIT Scripts: Black-Box Testing

4.1 SWIT Script Design

SWIT (Software Integration Test) scripts are like interface test scripts with the following extions:

• On the 532L project, the script tests an ORC (object relationship chart) as opposed to a ture chart. For non-object oriented 5300 products, this test will verify high level designs.

• The tests are performed on the target hardware instead of in a simulated environment suCodeCenter®.

SWIT Scripts: Black-Box Testing Page 15



ll

SWITtest-

d out-equiva-chnics,

300

his is

the

men-

• There are little or no stubs.

• Input and outputs are defined in Software Requirements Specification (e.g. Manpage). Athese inputs must be verified in the SWIT script

Related commands should be re-tested as well. For this example of adding a new user, thescript should contain tests where you login as the new user. For further information on SWITing , please refer to the TITAN 5300 SWIT Test Guidelines document[5].

4.2 Determining Test Cases

Again, the Software Requirements Specification should be used to determine the inputs anputs that need to be verified. In order to keep the number of tests on the inputs reasonable,lence class and boundary-values analysis can be used. For more information on these teplease refer to the Unit Test Guideline document[4].

4.3 The Standard

SWIT scripts must adhere to the following rules:

• On the 532L, there must be a SWIT script for each ORC. On other non-object oriented 5products, a SWIT script must be written for each high level design.

• Each test must provide full coverage of both the valid and invalid equivalence classes. Tshown by a list of all the inputs and outputs with a test step for each one.

• The SWIT script must have the same name as the ORC./high level design with “.swit” atend.

• The following template must be used for SWIT test scripts with requirements and recomdations listed

Requirements:

• Header Section

• Test Case Number

• Summary Section



• Date of approval at end of test

• Signature of Tester at end of test

Recommendations:

• All inputs and outputs listed at the beginning of the test script

/******************* BEGIN **************************









*******************************************************

*

* Name:<Name of Test>

* Type:SWIT Script

* Originator:<author>

* File:<filename>


* Reference: <High Level Design Reference>

* Description: <One Line Description>.

*

*

************************ END **************************

*/

Inputs:<List of Inputs for Test Script>

Outputs:<List of Outputs for Test Script>

Test 1.

Summary:

<Description of Test to Be executed>

Input:

<Inputs required for Test>

Expected Output:

<Expected outputs from Test>

Status:

<Pass or Fail>

•

•

•

Date:

Signature:




4.4 The SWIT Script for user_add on the TITAN 532L

The ORC foruser_add is shown in Figure 2.

FIGURE 2. The ORC for user_add

4.5 Manpage for UTL_ADD_USER

UTL::ADD,USER(1) 532L PDS Commands UTL::ADD,USER(1)

NAME

UTL::ADD,USER - Add a new user to the system

craft:add user

user:add

user list

username,group,default_password

username,group

ok




s case

SYNOPSIS

UTL::ADD,USER name, LEV level!

SECURITY LEVEL: Supervisor

DESCRIPTION

This command adds a new user to the system with user name

name and security level

RESPONSES

M time frame, seq cmd_link UTL ADD USER name LEV level COMPL

M time frame, seq cmd_link UTL ADD USER name LEV level error

DNY

where error is one of the following:

US-DUPUSER The user name already exists in the system.

US-LISTFULL The system already contains the maximum

number of users allowed.

SEE ALSO

utl_del_user(1)

Feature Package 1Last change: 14 September 1990 1

4.6 Inputs and Outputs for UTL_ADD_USER

The inputs and outputs are listed below. The outputs can be found in the man page, in thiutl_add_user(1) .

1. username < USNAMELEN

2. username == USNAMELEN




3. username > USNAMELEN

4. group == CRAFT

5. group == 6


7. userlist full

8. username not already in system

9. username already in system

a. COMPL

b. US-DUPUSER DNY

c.US-LISTFULL DNY

d.BDARG DNY

4.7 The Swit Script of UTL_ADD_USER

/******************* BEGIN **************************






*******************************************************

*

* Name:Add user

* Type:SWIT Script


* File:USadd.swit

* Version:1.1

* Reference: user_add.sce


*

*

************************ END **************************

*/

Inputs:

1. username < USNAMELEN

2. username == USNAMELEN




3. username > USNAMELEN

4. group == CRAFT

5. group == 6


7. userlist full

8. username not already in system

9. username already in system

Outputs:

a. COMPL

b. US-DUPUSER DNY

c. US-LISTFULL DNY

d. BDARG DNY

Test 1.Adding a User on a empty list

Input:

UTL::ADD,USER steve, LEV craft!

name = “steve”

group = CRAFT


Expected Output:

M time frame, seq cmd_link UTL ADD USER steve LEV craftCOMPL

Status: Pass

Test 2.Adding a user with name equal to maximum value

Input:

UTL::ADD,USER ANameWithTwentyChars, LEV craft!

name = “ANameWithTwentyChars”

group = CRAFT

userlist = “steve”, CRAFT

Expected Output:

M time frame, seq cmd_link UTL ADD USER steve LEV craftCOMPL

Status: Pass

Test 3.Adding a user with a long name

Input:

UTL::ADD,USER AnExtremelyLongNameThatExceedsUSNAMELEN,LEV craft!




name = “AnExtremelyLongNameThatExceedsUSNAMELEN”

group = CRAFT


“ANameWithTwentyChars”, CRAFT

Expected Output:

M time frame, seq cmd_link UTL ADD USER AnExtremely-LongNameThatExceedsUSNAMELEN LEV craft COMPL

Status: Pass

•

•

•

Test 6.Adding a duplicate user

Input:

UTL::ADD,USER steve, LEV craft!

name = “steve”

group = CRAFT


“ANameWithTwentyChars”, CRAFT

“AnExtremelyLongNamet”, CRAFT

Expected Output:

M time frame, seq cmd_link UTL ADD USER steve LEV craftUS-DUPUSER DNY

Status: Pass

Date: 1/1/90

Signature: S. Harpster



test script standards - purdue university - department …® 5300 software july 13, 1994 test script...

Documents