testcover.com copyright © 2005 testcover.com, llc. all rights reserved. the bug stops here...

Copyright © 2005 Testcover.com, LLC. All rights reserved.

Testcover.com

The Bug Stops HereThe Bug Stops Here

Improving Test Designwith a

Novel Test Case Generator

Technical PresentationSalon 6

Wednesday May 18, 1:45 – 2:45


Testcover.comHow to Spin a WebHow to Spin a Web

of Effective Test Casesof Effective Test Cases• Test case generator makes “all

pairs” testing simpler, quicker, more practical

• New technology improves test case coverage and fault discovery

• Service continually integrates research advances to reduce number of test cases


Testcover.comTesting Breakthrough Testing Breakthrough

NeededNeeded• Increasing dependency on software,

networks and distributed applications

• Increasing expectation and need for quality

• Facts of life:• More features and complexity• Shorter development cycles• More reuse of components


Testcover.com

This PresentationThis Presentation• Describes approaches to software test

case generation• Test configurations• System-level functional tests – integration

tests, functional verification tests, regression tests etc.

• Does not address• Unit testing (e.g. code coverage)• Stability and load testing


Testcover.com

This PresentationThis Presentation• Explains how the Testcover.com Covering

Test Case Generator Service improves test case coverage and fault discovery

• Shows how to use the service with examples• Tells how to receive 1 month service free• Is available on the web at Testcover.com


Testcover.com

• Founded in 2003 to aid testers in the selection of efficient sets of test cases

• Covering Test Case Generator Service• On the web at Testcover.com• Technology based on:

• advances in covering array construction• practical software engineering

experience

Testcover.com Dedicated to Excellence in Test

Design


Testcover.com

Configuration ExampleConfiguration Example

Application BrowserDisplay

ResolutionOperating System Connection

Application 1Internet Explorer High XP Dial-up

Application 2 Netscape Medium ME DSL

Application 3 Mozilla Low Cable

Firefox LAN

• Test three new applications on a networked computer• Test factors and their values:


Testcover.com

Configuration ExampleConfiguration Example• A simple approach:

• One test configuration for each factor value

• Select a most common or default value for each factor

• Vary each factor over all its values while the other factors are set to their default values


Testcover.com

Configuration ExampleConfiguration Example




Default Values



Firefox LAN

= 3 + 4 + 3 + 2 + 4 = 16Duplicate test cases: - 4

________= 12

• A simple approach:• How many test configurations? 12


Testcover.com

Configuration ExampleConfiguration Example• A simple approach:

• Small number of test configurations• All individual factor values are covered• Many pairs of factor values are not:

• Application 2 with Cable Connection• Firefox Browser with ME Operating System• Application 3 with Medium Resolution• More…


Testcover.com

Configuration ExampleConfiguration Example• A better approach:

• Include all pairs of factor values• Test all pair-wise interactions

• How many pairs of factor values? 101• How many test configurations? 16


Testcover.com






Firefox LAN

= 3 x 4

Configuration ExampleConfiguration Example How many pairs of factor values?


Testcover.comConfiguration ExampleConfiguration Example

How many pairs of factor values?






Firefox LAN

= 3 x 4 + 3 x 3


Testcover.com






Firefox LAN

= 3 x 4 + 3 x 3 + 3 x 2



Testcover.com






Firefox LAN

= 3 x 4 + 3 x 3 + 3 x 2 + 3 x 4 = 39



Testcover.com






Firefox LAN

= 3 x 4 + 3 x 3 + 3 x 2 + 3 x 4 = 39

= 4 x 3



Testcover.com






Firefox LAN

= 3 x 4 + 3 x 3 + 3 x 2 + 3 x 4 = 39

= 4 x 3 + 4 x 2



Testcover.com






Firefox LAN

= 3 x 4 + 3 x 3 + 3 x 2 + 3 x 4 = 39

= 4 x 3 + 4 x 2 + 4 x 4 = 36



Testcover.com






Firefox LAN

= 3 x 4 + 3 x 3 + 3 x 2 + 3 x 4 = 39= 4 x 3 + 4 x 2 + 4 x 4 = 36

= 3 x 2



Testcover.com






Firefox LAN

= 3 x 4 + 3 x 3 + 3 x 2 + 3 x 4 = 39= 4 x 3 + 4 x 2 + 4 x 4 = 36

= 3 x 2 + 3 x 4 = 18



Testcover.com






Firefox LAN

= 3 x 4 + 3 x 3 + 3 x 2 + 3 x 4 = 39= 4 x 3 + 4 x 2 + 4 x 4 = 36

= 3 x 2 + 3 x 4 = 18

= 2 x 4 = 8+_______

101



Testcover.com

Configuration ExampleConfiguration Example• 16 configurations cover all 101 pairsTest Combo

Case ID Countdown

3 Values 4 Values 3 Values 2 Values 4 Values 101

1 App1 Netscape High ME LAN 91

2 App3 Firefox Medium ME Dial-up 81

3 App2 Mozilla Low ME Cable 71

4 App2 Firefox High XP DSL 61

5 App1 IE Low XP LAN 52

6 App3 Netscape Medium XP Cable 43

7 App1 Mozilla Medium ME DSL 36

8 App3 IE High ME Cable 30

9 App2 Mozilla High XP Dial-up 24

10 App3 Netscape Low ME DSL 19

11 App2 IE Medium ME DSL 15

12 App3 Mozilla Medium XP LAN 11

13 App1 Firefox Low ME Cable 7

14 App2 Netscape Low ME Dial-up 4

15 App1 IE High XP Dial-up 2

16 App2 Firefox Low ME LAN 0

Display Resolution Operating System ConnectionApplication Browser


Testcover.com

Combo CountdownCombo Countdown• Coverage metric• Number of untested pairs after each

test case• Stops at 0 or the number of

untested pairs when there are constraints

• Illustrates diminishing returns• Supports risk assessment


Testcover.com

Test Test DefinitionsDefinitions

• Test factor – any variable whose values are to be controlled during testing

• Test factor value – specific value taken by one of the test factors

• Test case – set of test factor values with one allowed value for each of the test factors

• Combination – association of some number of factor values (pairs, triples, etc.)


Testcover.com

Array DefinitionsArray Definitions• Size – number of test cases,

number of rows in array• Degree – number of test factors,

number of columns in array• Order – number of values for a test

factor• Strength – size of combinations

(2: pairs, 3: triples, etc.)


Testcover.com

Mixed arraysMixed arrays• Mixed array has factors with different

orders• Example: 52 43 32 21

• 2 factors with 5 values• 3 factors with 4 values• 2 factors with 3 values• 1 factor with 2 values

• Degree = 2 + 3 + 2 + 1 = 8• There is a 52 43 32 21 array with strength 2 and

size 25


Testcover.com

Early Early ApproachesApproaches

• Orthogonal arrays used in agricultural and medical experiments

• Dr. Genichi Taguchi used orthogonal arrays to select test parameters

• Statistical coverage where exhaustive testing is not possible


Testcover.com

Real-world ConstraintsReal-world Constraints• What if the applications in the

configuration example are ported to Linux and Macintosh OSX?

• Internet Explorer is supported on Windows only

• There are Browser-Operating System pairs that must be excluded


Testcover.com

Configuration ExampleConfiguration Example• The wrong answer:

Test Combo

Case ID Countdown

3 Values 4 Values 3 Values 4 Values 4 Values 129

1 App1 IE High XP Dial-up 119

2 App2 IE Medium ME DSL 109

3 App3 IE Low Linux Cable 99

4 App1 Netscape Medium Linux LAN 89

5 App2 Netscape High MacOSX Cable 79

6 App3 Mozilla High ME LAN 69

7 App1 Mozilla Low MacOSX DSL 59

8 App2 Firefox Low XP LAN 49

9 App3 Firefox Medium MacOSX Dial-up 39

10 App3 Netscape Low XP DSL 32

11 App2 Mozilla Low Linux Dial-up 25

12 App1 Firefox Low ME Cable 18

13 App3 Mozilla Medium XP Cable 12

14 App3 Firefox High Linux DSL 6

15 App3 IE Low MacOSX LAN 3

16 App3 Netscape Low ME Dial-up 0

Application Browser Display Resolution Operating System Connection


Testcover.comConstrained Array Test Constrained Array Test

SystemSystem• Motivated by testing Local Area Network

product configurations• Developed in 1990 to handle constraints• Employed greedy search algorithm• Used internally at AT&T as best practice• Described in STAR proceedings May 1994


Testcover.com

Current ChallengesCurrent Challenges

1. Find small array quickly2. Adapt to system constraints simply3. Support multiple system states4. Allow easy use

Testcover.com addresses these challenges

Let’s see how…


Testcover.com

Sample ArraysSample Arrays

Size Array Size Array Size Array

6 210 23 48 61 710

8 230 28 424 91 730

11 35 33 58 78 811

15 320 45 530 120 830

19 330 46 69 81 910

16 45 76 627 153 930


Testcover.com

Sample Mixed ArraysSample Mixed Arrays

Size Degree

Array Size Degree

Array

15 30 36 224 12 10 41 33 26

22 25 45 220 20 17 51 44 212

30 25 45 320 25 8 52 43 33

27 23 53 320 30 30 61 55 224

39 23 63 320 36 15 62 51 312

49 23 73 420 42 15 71 62 312


Testcover.com

Challenge 1: Fewer Test Challenge 1: Fewer Test CasesCases

• Small arrays result from multiple techniques

• Testcover.com continually integrates research advances to reduce the numbers of test cases

• New technology provides quicker results

The results given here are subject to improvement without notice!


Testcover.comChallenge 2: Handle Challenge 2: Handle

ConstraintsConstraints• Direct Product Block notation

enables simple descriptions of complex constraints and dependencies

• Let’s see how DPB notation works with the Configuration Example


Testcover.com

Configuration ExampleConfiguration Example• No constraints

– one block

Configuration Example

Application

Browser

Display Resolution

Operating System

Connection

#

App1 App2 App3

IE Netscape Mozilla Firefox

High Medium Low

XP ME

Dial-up DSL Cable LAN


Testcover.comDirect Product Block Direct Product Block

NotationNotation• Block represents all possible test

cases for selected values of the test factors

• Multiple blocks represent all allowed test cases – as a set of products

• Simpler than enumerating all disallowed combinations


Testcover.com

Configuration ExampleConfiguration Example• Browser/OS

constraints – two blocks

Configuration Example - With Constraints

Application

Browser

Display Resolution

Operating System

Connection

#

App1 App2 App3

IE Netscape Mozilla Firefox

High Medium Low

XP ME


+

App1 App2 App3

Netscape Mozilla Firefox

High Medium Low

Linux MacOSX



Testcover.comChallenge 3: Multiple Challenge 3: Multiple

StatesStates• Calendar example illustrates:

• Direct Product Block notation• Constraints – different month lengths• Multiple states using different

partitions of test cases –• Normal (good dates)• Error handling (too long months)


Testcover.comCalendar Calendar ExampleExample

Request Part 1

Calendar Example

Month

Day

Year

#ok All good dates

jan feb mar apr may jun jul aug sep oct nov dec

1 10

2003 2004 2005

+ long month last day

jan mar may jul aug oct dec

31

2003 2004 2005

+ short month last day

apr jun sep nov

30

2003 2004 2005

+ feb last day

feb

28

2003 2005

+ leap day

feb

29

2004



Request Part 2

#err Too long month

+ too long long month

jan mar may jul aug oct dec

32

2003

+ too long short month

apr jun sep nov

31

2004

+ feb too long, regular year

feb

29

2005

+ feb too long, leap year

feb

30

2004



Results Part 1

Test Combo

Case ID Countdown

12 Values 6 Values 3 Values 126

ok1 jan 1 2003 123

ok2 jan 10 2004 120

ok3 feb 10 2003 117

ok4 feb 1 2005 114

ok5 mar 1 2004 111

ok6 mar 10 2005 108

ok7 may 31 2005 105

ok8 aug 31 2004 102

ok9 oct 31 2003 99

ok10 nov 30 2004 96

ok11 apr 30 2005 93

ok12 jun 30 2003 90

ok13 feb 29 2004 87

ok14 apr 1 2003 85

ok15 apr 10 2004 83

ok16 may 10 2003 81

ok17 jun 1 2004 79

ok18 jun 10 2005 77

ok19 jul 1 2003 75

ok20 jul 10 2004 73

YearMonth Day

#1. All good dates



Results Part 2

Test Combo

Case ID Countdown


ok21 aug 10 2003 71

ok22 aug 1 2005 69

ok23 sep 10 2003 67

ok24 sep 1 2004 65

ok25 oct 10 2004 63

ok26 nov 10 2003 61

ok27 nov 1 2005 59

ok28 dec 10 2003 57

ok29 dec 1 2004 55

ok30 jul 31 2005 53

ok31 dec 31 2005 51

ok32 jan 31 2005 49

ok33 mar 31 2003 47

ok34 sep 30 2005 45

ok35 feb 28 2003 43

ok36 may 10 2004 42

ok37 may 1 2005 41

ok38 oct 1 2003 40

ok39 oct 10 2005 39

ok40 feb 28 2005 38

YearMonth Day



Results Part 3

Test Combo

Case ID Countdown


err1 jan 32 2003 93

err2 apr 31 2004 90

err3 feb 29 2005 87

err4 feb 30 2004 84

err5 mar 32 2003 82

err6 may 32 2003 80

err7 jul 32 2003 78

err8 aug 32 2003 76

err9 oct 32 2003 74

err10 dec 32 2003 72

err11 jun 31 2004 70

err12 sep 31 2004 68

err13 nov 31 2004 66

Month Day Year

#2. Too long month


Testcover.com

Challenge 4: Ease of UseChallenge 4: Ease of Use• Direct Product Block notation

• Simple to learn• Flexible and efficient

• Internet-based service• Easy access without special installation• Independent of development

environment• Enhancements without upgrade work

• Practical for everyday use


Testcover.com

Challenge 4: Ease of UseChallenge 4: Ease of Use• Advanced Web Search example

illustrates:• Practical for real HTML form• Direct Product Block notation• Constraints – language button use• Small number of test cases


Testcover.comAdvanced Web Search Advanced Web Search

ExampleExample• See HTML form


Testcover.comAdvanced Advanced

Web Search Web Search ExampleExample

• Request part 1

Y! Advanced Search Form

All words

All page part

Exact phrase

Exact page part

Any words

Any page part

Excluded words

Excluded page part

Updated

Site/domain

Format

Filter

Country

Language

Arabic

Chinese

English

Japanese

Russian

Thai

Results/page




• Request part 2

#+any language

all list 1,all list 2,all list 3,all list 4any,title,url

exact phrase 1,exact phrase 2,exact phrase 3,exact phrase 4any,title,urlany list 1,any list 2,any list 3,any list 4

any,title,urlexcluded list 1,excluded list 2,excluded list 3,excluded list 4

any,title,urlanytime,past 3 months,past 6 months,past year

any,.com,.edu,.gov,.org,domain 1,domain 2all,.html,.pdf,.xls,.ppt,.doc,.xml,.txton,off

any,Argentina,Brazil,Denmark,France,Germany,United Statesany

uncheckedunchecked

uncheckeduncheckedunchecked

unchecked10,15,20,30,40,100




• Request part 3

+specified language(s)

all list 1,all list 2,all list 3,all list 4

any,title,url

exact phrase 1,exact phrase 2,exact phrase 3,exact phrase 4

any,title,url

any list 1,any list 2,any list 3,any list 4

any,title,url

excluded list 1,excluded list 2,excluded list 3,excluded list 4

any,title,url

anytime,past 3 months,past 6 months,past year

any,.com,.edu,.gov,.org,domain 1,domain 2

all,.html,.pdf,.xls,.ppt,.doc,.xml,.txt

on,off

any,Argentina,Brazil,Denmark,France,Germany,United States

specified

checked,unchecked

checked,unchecked

checked,unchecked

checked,unchecked

checked,unchecked

checked,unchecked

10,15,20,30,40,100


Testcover.comAdvanced Web Search Advanced Web Search

ExampleExample• 81 72 61 45 34 28 Array• Degree 21• 2715 pairs, 6 not covered

(constraints)• Size 63 (compared to 8 x 7 = 56)


Testcover.comHow to Spin a WebHow to Spin a Web

of Effective Test Casesof Effective Test Cases• New technology improves test

case coverage and fault discovery• Service continually integrates

research advances to reduce number of test cases

• Exclusive Direct Product Block notation simplifies use


Testcover.com

Exhibit at Booth 8Exhibit at Booth 8• See these and other examples run• Receive your brochure and gift• Register for 1 free month of

service


Testcover.comSTAREAST 2005 First Month STAREAST 2005 First Month

FreeFree4 Month Subscription for the Price of 3

• For STAREAST 2005 Attendees Only• Register at Booth 8 May 18 – May 19• Try Testcover.com Service Free for 1

Month• Your Choice:

• Pay invoice for 4-Month Subscription• Cancel On-line

• 1 Month Free Service either way – $100 Value


Testcover.com

Back-up SlidesBack-up Slides


Testcover.com

Test Case ObjectivesTest Case Objectives• Run enough of the “right” test cases

to exercise system interactions completely

• Avoid wasting time and money on unnecessary test cases

• Insure system behavior is thoroughly verified

• Allow for complex and changing system requirements


Testcover.comOrthogonal Arrays & Covering Orthogonal Arrays & Covering

ArraysArrays• Every pair of columns in an orthogonal

array contains all pairs of values an equal number of times

• Every pair of columns in a covering array contains all pairs of values at least once

• Orthogonal arrays are covering arrays• Both types can be mixed (different

orders)


Testcover.comOrthogonal Arrays & Covering Orthogonal Arrays & Covering

ArraysArrays• Orthogonal arrays are balanced –

suggest which factor values give best measured results• Tuning complex systems• Medical & agricultural trials

• Covering arrays are not required to be balanced – can be smaller for pass/fail type tests


Testcover.comCovering Arrays vs. Random Covering Arrays vs. Random

ValuesValues• Goal: Validate operation quickly –

integration, function, regression – Arrays provide coverage with few test cases

• Goal: Validate continuous operation – stability, load, stress – Random values can provide many different test cases


Testcover.com

• Strength t arrays are defined similarly – Every set of t columns in an array contains all t-tuples of values (3-tuples, 4-tuples, etc.)

• Size (number of test cases) increases with strength – Minimum size is product of the t largest orders

• Example: 52 43 32 21 arrays

Strength: 2 3 4

Minimum Size: 25 100

400

Higher Strength ArraysHigher Strength Arrays


Testcover.com

Higher Strength ArraysHigher Strength Arrays• Higher strength arrays can cover more

combinations of factor values, with a cost of more test cases

• Generally finding higher strength arrays requires more processing

See Sherwood, Martirosyan, and Colbourn, Sections 4 & 5.


Testcover.com

RDF/XML ResultsRDF/XML Results• The Resource Description Framework (RDF) is a general-

purpose language endorsed by the World Wide Web Consortium (W3C) for representing information on the web.

• RDF is based on the idea of identifying things using Uniform Resource Identifiers (URIs) and describing resources in terms of simple properties.

• RDF represents simple statements about resources as a graph of nodes and arcs representing the resources, and their properties and values.

• RDF also provides an XML-based syntax (called RDF/XML) for recording and exchanging these graphs.

• In addition to HTML results, the Testcover.com test case generator provides RDF/XML results for use by other applications.


Testcover.comSources & Further Sources & Further

ReadingReadingCompanies Seek to Hold Software Makers Liable for Flaws, David Bank, The Wall Street Journal, page B1, February 24, 2005.

To Darned Big to Test, Keith Stobie, ACM Queue, vol. 3, no. 1, pages 30-37, February 2005.

Products of Mixed Covering Arrays of Strength Two, C. J. Colbourn, S. S. Martirosyan, G. L. Mullen, D. Shasha, G. B. Sherwood, and J. L. Yucas, preprint, 2004.

Covering Arrays of Higher Strength From Permutation Vectors, G. B. Sherwood, S. S. Martirosyan, and C. J. Colbourn, preprint, 2004.

On the Construction of Orthogonal Arrays and Covering Arrays Using Permutation Groups, George Sherwood, http://home.att.net/~gsherwood/cover.htm March 2002 - June 2004.


Testcover.comSources & Further Sources & Further

ReadingReadingEffective Testing of Factor Combinations, George Sherwood,

Third International Conference on Software Testing, Analysis & Review, May 8-12, 1994, Washington, DC.

Orthogonal Arrays Theory and Applications, A. S. Hedayat, N. J. A. Sloane, and John Stufken, 1999 Springer-Verlag.

Quality Engineering Using Robust Design, M. S. Phadke, 1989, Prentice Hall.

M. J. Suárez-Cabal and J. Tuya. Using an SQL Coverage Measurement for Testing Database Applications. ACM Software Engineering Notes. 29(6):253-262, 2004.

testcover.com copyright © 2005 testcover.com, llc. all rights reserved. the bug stops here...

Documents

test design slide

configuration example

test case coverage

test case generator

number of test cases

pairs of factor values

load testing slide

individual factor values