Chin-Yu HuangDepartment of Computer Science

National Tsing Hua UniversityHsinchu, Taiwan

Optimal Allocation of Testing-Resource Considering Cost, Reliability, and Testing-

Effort

The 10th International Symposium Pacific Rim Dependable Computing

(PRDC 2004)

2

Outline Introduction Software Reliability Growth Model with

Generalized Logistic Testing-Effort Function Testing Resource Allocation for Software

Module Testing Experimental Studies and Results —Numerical Example and Sensitivity

Analysis Conclusions

3

Introduction A software testing process consists of several

testing stages including module testing, integration testing, system testing, and installation testing.

The quality of the tests usually corresponds to the maturity of the software test process, which in turn relates to the maturity of the overall software development process.

Most popular and commercial software products are complex systems and composed of a number of modules.

4

Introduction (contd.)

Project managers should know how to allocate the specified testing-resources among all the modules and develop quality software with high reliability.

We provide a systematic method for the project managers to allocate specific amount of testing-resource expenditures for each module under some constraints: minimizing the cost of software development, with a given fixed amount of testing-effort and a reliability objective.

5

SRGMs Software Reliability represents a customer-oriente

d view of software quality and is dynamic rather than static.

SRGMs describe failures as a random process, which is characterized in either times of failures or the number of failures at fixed time.

We use an SRGM with generalized logistic testing-effort function to describe the time-dependency behaviors of detected faults and the testing-resource expenditures spent during module testing.

6

Generalized Logistic Testing-Effort Function The generalized logistic testing-effort function (LTEF) was proposed by Huang, Kuo, and Lyu (1999) an

d can be depicted as

where N is the total amount of testing effort to be eventually consumed, a is the consumption rate of testing-effort expenditures, A is a constant, and k is a structuring index with a large value for modeling well-structured software development efforts.

tAe

NtW

1

7

SRGM with Generalized Logistic Testing-Effort Function

))])()((exp[()( 0WtWr1atm ))]((exp[ tWr1a

where m(t) is the expected mean number of faults detected in time (0, t) , Wk(t) is the current testing-effort consumption at time t, a is the expected number of initial faults, and r is the error detection rate per unit testing-effort at testing time t & r>0.

This SRGM with Generalized LTEF was proposed by Huang, Kuo, and Lyu (1999) and the mean value function is

8

SRGM with Generalized Logistic Testing-Effort Function (contd.)

When t→∞, the expected number of faults to be detected is

The reliability of a software system is defined as the ratio of the cumulative number of detected faults at time t to the expected number of initial faults:

aA

Nam

]

1exp[1)( (if A >> N)

))(exp(1)(

)( trWa

tmtR

9

Testing-Resource Allocation for Module Testing

Assumptions :

1) The software system is composed of N independent software modules that are tested individually. The number of faults remaining in each module can be estimated by an SRGM with generalized LTEF.

2) For each module, the failure data have been collected and the parameters of each module can be estimated.

3) The total amount of testing resource expenditures available for the module testing processes is fixed and denoted by W.

10

Testing-Resource Allocation for Module Testing

Assumptions (contd.):

4) If any of the modules is faulty, the whole software system fails.

5) The system manager has to allocate the total testing resources W to each software module to minimize the number of faults remaining in the system during the testing period.

6) The desired software reliability after the testing phase is greater than or equal to the reliability objective R0.

11

Mean Value Function of a Software System with N Modules

The number of detected faults in the system can be estimated by

where vi is a weighting factor to measure the relative importance of a fault removal from module i in the future.

If vi =1 for all i =1, 2,…, N, the objective is to minimize the total number of faults remaining in the software system after the testing phase.

N

iiiii

N

iii tWravtmv

11)))(exp(1()(

12

Mean Value Function of a Software System with N Modules (contd.)

Therefore, the number of remaining faults can be estimated by

N

iiiii

N

iii WravtWrav ii

11)exp())(exp(

13

Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective

The objective function is : minimizing the cost of software testing

Subject to the constraints : (1) The total amount of testing resource expenditures for all modules would not be more than available resource W. (2) Reliability of each module should not be less than R0 guarantee that reliability of system will not be less than R0 .

14

Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective

The objective function is:

Minimize:

Subject to the constrains:

, i=1, 2,..., N.

)(1

N

iii WCost

0,1

i

N

ii WWW

0))(exp(1)( RtWrtR iii

15

Therefore, we have:

, i= 1,2,…,N )1ln(1

0Rr

Wi

i

)1ln(1

0Rr

Di

i

Let

),.....,,,,0max( 321 NDDDDC

We can have , ,i=1,2,….,N

N

ii WW

1

0iW

and ,whereCWi

Let , we can transform the above equations to

The following two equation:

CWX ii

Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective

16

Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective

Minimize:

Subject to

)(*1

N

iii XCost

0,11

i

N

ii

N

ii XCWX ,i=1,2, ..., N.

where

N

iiiii

N

iii CravCXCost

11 )exp(1(*)(*

1

N

iiiiiiiii XrCravCXr

12 )exp()exp(*))exp(

N

iii CXC

13 )(*

17

According to the Lagrange multiplier, above equationsbe simplified as follows:

Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective

Minimize:

N

iiiiiiiN XrCravCXXXL

11 ))exp()exp(1(*),,...,,( 21

)exp()exp(*1

2 iiiiii XrCravCN

i

)()(*111

3

N

ii

N

iiii CWXCXC

N

i

18

Based on the Kuhn-Tucker conditions (KTC) , the necessary conditions for a minimum value of above equation are in existence .

The results will be applied in the Algorithm for solving our problem.

Minimizing the Software Cost with a Given Fixed Amount of Testing-Effort and a Reliability Objective

19

Algorithm 1Step 1: Set l=0.

Step 2: Calculate the following equations

)}*ln())exp()**({ln(1

312 CCrCCravr

X iiiiii

i ,i=1, 2,..., N- l.

3*C

N

ii

N

i

N

iiiiiiii

r

CWCrCCravr

1

1 1

)/1(

))exp()**()(ln/1(exp

12

Step 3: Rearrange the index i such that .... ***21 lNXXX

Step 4: IF then0* lNX stop (i.e., the solution is optimal)0* lNXElse ; l=l+1

End-IF.Step 5: Go to Step 2.

20

Observation 1

The optimal solution has the following form :

otherwiseX

r

CWCrCCravr

Cwhere

lNi

CCrCCravr

X

i

N

ii

N

i

N

iiiiiiii

iiiiii

i

,0

)/1(

))exp()**()(ln/1(exp

*

.,...,2,1,

)}*ln())exp()**({ln(1

*1

1 1

*

12

3

312

21

Observation 2 Algorithm 1 always converges in, at worst, N-1 steps and thus the

value of objective function at the optimal solution is

),...,,( **2

*1 NXXX

N

iiiiiii CXrCravC

12

*1 *))exp()exp(1(*

N

ii

N

iiiiiii CXCXrCrav

1

*13

1

* )(*)exp()exp(

22

Numerical Examples Suppose the total amount of testing-effort expendit

ures W is given.

We apply the proposed model to actual software failure data and have to allocate the expenditures to each module to minimize the expected cost.

All the parameters ai and ri for each software module have been estimated by using the maximum likelihood estimation (MLE) or the least squares estimation (LSE).

23

Table 1 : the estimated values of ai , ri , vi , and .Module ai ri vi

1 89 4.1823×10-4 1 1.0 7632

2 25 5.0923×10-4 1 0.6 3158

3 27 3.9611×10-4 1 0.7 4009

4 45 2.2956×10-4 1 0.4 4329

5 39 2.5336×10-4 1 1.5 8964

6 39 1.7246×10-4 1 0.5 4568

7 59 8.819×10-5 1 0.5 6023

8 68 7.274×10-5 1 0.6 9112

9 37 6.824×10-5 1 0.05 0

10 14 1.5309×10-4 1 1 2203

*iX

An example for Proposed Algorithm

24

Sensitivity analysis

If a1 is increased by 40%, then the estimated value of optimal testing-effort expenditure for module 1 is changed from 7632 to 8400 and its relative change (RC) is 0.100628931 about 10% increment).

for modules 2, 3, 4, 5, 6, 7, 8, and 10, the estimated values of optimal testing-effort expenditures are decreased by about 0.95%, 0.95%, 1.52%, 0.67%, 1.93%, 2.87%, 2.30%, and 4.49%, respectively.

25

Sensitivity analysis (contd.)

1 2 3 4 5 6 7 8 9 10

-0.04

-0.02

0.00

0.02

0.04

0.06

0.08

0.10

Re

lativ

e C

ha

ng

e

Modules

a1*1.4

a1*1.3

a1*1.2

a1*1.1

Figure 1: Relative change of OTEE for the case of 40%, 30%, 20%, and 10% increase to a1.

26

If a1 is decreased by 30% the estimated value of optimal testing-effort expenditure for module 1 is changed from 7632 to 6818 and its RC is 0.106656184 (about 10.66% decrement). for modules 2, 3, 4, 5, 6, 7, 8, and 10, the estimated values of optimal testing-effort expenditures are increased by about 1.01%, 1.02%, 1.64%, 0.71%, 2.06%, 3.05%, 2.44%, and 4.86%, respectively.

Sensitivity analysis (contd.)

27

Sensitivity analysis (contd.)

1 2 3 4 5 6 7 8 9 10

-0.16

-0.14

-0.12

-0.10

-0.08

-0.06

-0.04

-0.02

0.00

0.02

0.04

0.06

0.08

0.10R

ela

tive

Ch

an

ge

Modules

a1*0.6

a1*0.7

a1*0.8

a1*0.9

Figure 2: Relative change of OTEE for the case of 40%, 30%, 20%, and 10% decrease to a1.

28

Sensitivity analysis (contd.)

If a1 & a2 both are increased by 40%

the estimated values of optimal testing-effort expenditure for modules 1 and 2 are changed from 7632 to 8370 (about 9.67% increment) and 3158 to 3764 (about 19.18% increment), respectively.

for modules 3, 4, 5, 6, 7, 8, and 10, the estimated values of optimal testing-effort expenditures are decreased by about 1.75%, 2.79%, 1.23%, 3.52%, 5.25%, 4.19%, and 8.22%, respectively.

29

Sensitivity analysis (contd.)

Figure 3: Relative change of OTEE for the case of 40%, 30%, 20%, and 10% increase to a1 & a2.

1 2 3 4 5 6 7 8 9 10

-0.10

-0.08

-0.06

-0.04

-0.02

0.00

0.02

0.04

0.06

0.08

0.10

0.12

0.14

0.16

0.18

0.20

0.22R

ela

tive

Ch

an

ge

Modules

a1*1.4 & a

2*1.4

a1*1.3 & a

2*1.3

a1*1.2 & a

2*1.2

a1*1.1 & a

2*1.1

30

Conclusions In this paper, we proposes a method to optimize

the software testing-resource allocation problem —minimizes the cost of software development, with a given fixed amount of testing-effort and a reliability objective.

We develop a comprehensive strategy for module testing in order to help software project managers make the best decisions in practice.

31

Conclusions (contd.)

An extensive sensitivity analysis is presented to study the effects of various principal parameters on the optimization problem of testing-resource allocation.

Using Algorithm 1, project managers can allocate limited testing-resource easily and efficiently and thus achieve the lowest cost objective during software module and integration testing.