1

Model for Optimizing Software Testing Period using Non Homogenous Poisson Process based on Cumulative Test Case Prioritization

Praveen Ranjan Srivastava

Computer Science and Information Systems Group Birla Institute of Technology and Science (BITS)

Pilani-333031 INDIA

praveenrsrivastava@gmail.com

Abstract- Most of the software organizations has trouble when deciding the release dates their product. This difficulty is due to the fact that an under tested software could lead to many bugs propping up at the client side which would in turn lead to expensive bug-fixes and more importantly loss of customer goodwill. On the other hand, testing beyond certain time would lead to loss of revenue to the organization due to the dilution of the early bird advantage. The aim of our paper is optimizes the time and cost of entire software. In this paper we used non homogeneous Poisson process model based on cumulative priority. Our paper also tries to answer when to release any software. Key-Words: - Non Homogenous Poisson Process, Cumulative Priority, Software Life Cycle Length, Software Release Time, Optimal Testing Policy. 1. Introduction

Software systems are omnipresent. It is not infrequent that the life of people depends on the frictionless and reliable functioning of the programs; a point in case is medical technology. Professional verifying and testing of software therefore becomes an increasingly important task which requires well-founded education. Testing is a process used to help identify the correctness, completeness and quality of developed computer software [1]. With that in mind, testing can never completely establish the correctness of computer software. Software testing is a strenuous and expensive process. Research has shown that at least 50% of the total software cost is comprised of testing activities [2]. Companies are often faced with lack of resources, which limits their ability to effectively complete testing efforts There are many approaches to software testing, but effective testing of complex products is essentially a process of investigation, not merely a matter of creating and following rote procedure. According to the White box approach of testing, software is viewed as a white-box, or glass-box, and the structure and flow of the software under test are visible to the tester. Testing plans Praveen Ranjan Srivastava, doing research in the area of software testing.

are made according to the details of the software implementation, such as programming language, logic, and styles. Testing modules is potentially endless. We cannot test till all the bugs are removed - it is simply impossible. At some point, we have to stop testing and ship the software. The question is when. McDaid and Wilson proposed three plans to decide how long to test software [3]. The Stochastic and Economic Model Dalal and Mallows gave an idea to determine when to stop testing [4]. Musa and Ackerman quantified software validation based on the concept of reliability [5]. Ehrlich et al. proposed a model to determine the cost of a stop-test decision [6]. One of the most admired of these is the model proposed by Amrit L. Goel and Kazuhira Okumoto in November 1980[7]. They gave a cost based model based on time dependent fault detection rate model of Goel and Okumoto, and also described a policy that yields the optimal value of test time T [8, 9].

In order to eradicate the danger of not testing the crucial components of software, we introduce the concept of assigning priority weights. 2. Non-homogenous Poisson Processes

A Poisson process is a stochastic process which is used for modeling random events in time that occur to a large extent independently of one another [10]. Possible examples of such events include telephone calls arriving at a switchboard or webpage requests on a server. A homogeneous Poisson process is characterized by a rate parameter λ, also known as intensity, such that the number of events in time interval (t, t + τ] follows a Poisson distribution with associated parameter λτ. In probability theory, a non-homogeneous Poisson process is a Poisson process with rate parameter λ (t) such that the rate parameter of the process is a function of time. It’s a process whose behaviour is non-deterministic in that a state and does not fully determine its next state. This Process is used for modeling random events in time that occur to a large extent independently of one another. One example of such event includes, patients visiting a doctor. Another example of a non-homogeneous Poisson process would be the arrival rate of customers to a

2

restaurant over a day. The restaurant would notice the arrival rate increase during the meal hours and slow down during other parts of the day. The Poisson process is a collection {N (t): t ≥ 0} of random variables, where N (t) is the number of events that have occurred up to time t (starting from time 0). 3. A Review of the Goel and Okumoto Model 3.1. Time dependent model

A software system failure also occurs at random times [8, 11], caused by the faults present in the system. N (t) is the cumulative number of faults detected by time t (t>0). Let m (t) be the projected no. of software faults detected by time t. Goel and Okumoto have shown that for software failure processes, m (t) can be described by the equation

m (t) = a (1 – e-bt ) (1) Here ‘a’ represents the expected no. of software

failures to be eventually encountered and ‘b’ is the detection rate for an individual fault. The parameters a and b can be estimated from the available failure data which can either be in the form of times between failures or as cumulative failures during specified time intervals. 3.2. Cost Model and Testing Policy

According to Goel and Okumoto [11], the cost of fixing a fault during testing is generally much less than during the operation. However, the time spent in testing delays the software release, leading to increased cost overheads. The objective of the cost model is to reduce the test time and release the product early by determining the optimal release time. Goel and Okumoto gave the parameters c1, c2, c3, t and T which are as follows:

c1= cost of fixing a fault during testing c2= cost of fixing a fault during operation

(c2>c1) c3= cost of testing per unit time t = time spent in testing the product T= software release time (same as testing time) Since m (t) represents the expected number of faults

during (0, t), the expected costs of fixing faults during the testing and operational phases are c1m (T) and c2 (m (t)-m (T)), respectively. Further, the testing cost during a time period T is c3T. Combining these costs, the total expected cost is given by

C (T) = c1m (T) + c2 (m (t)-m (T)) + c3T (2) The optimal test policy based on average cost criteria

minimizes the average cost and depends on the ratio of ab and cr=c3/ (c2-c1) [11]. According to Goel and Okumoto, if a.b > Cr

T* = min (T0, t) (3)

Where, T0 = 1/b ln (a.b/ Cr). If the cost of delay in release is very high, the Goel and Okumoto model suggests no testing at all. 4. The Proposed Approach: Component Prioritization for Testing

Every software product ordered by the client has to go through the design phase. In the design phase the product is divided into modules. According to the model proposed by Srivastava and others in [10], those modules are further grouped based on the requirements, complexity and error vulnerability of the code.

This is not a significant deterrent as to reduce the cost of regression testing, most of the software testers prioritize their test cases so that those which are more important, by some measure, are run earlier in the regression testing process. Test case prioritization (TCP)[12] [13] involves execution of test cases in a pre-defined order so that the rate of fault detection early in the development cycle is improved, allowing the developers to rectify the faults earlier while improving the system delivery speed. Table 1: Priority Types Associated With Priority

Weights Priority Weight Priority Type

1 Very High

2 High 3 Medium 4 Low 5 Very Low

According to the model proposed in [9], a priority scale is developed with priorities ranging from very high to very low. Each priority level is associated with a ‘priority weight’. The above table shows a sample allotment of weights to different modules. The priorities would be allotted on the basis of the cumulative score which would be calculated by the formula shown below: CS = eCR * KLOC * CC * CP / KLOCavg Where CS = cumulative score,

CR = component rank as defined in part 5, KLOC = kilo lines of code, CC = cyclometric complexity, CP = customer priority (assigned a default value if not specified by the customer) KLOCavg = average of KLOC of all the modules in the project

The calculation of CS depends on all the factors stated above. Now, consider a situation where in there exist 5 modules in a software project and the modules 2, 3, 4&5 depend on module number 1. In such a case, there must be a considerable difference in the Component Score of the first module when compared with the rest of the modules. Clearly, the CR of that particular module is greater than the other four. Therefore to show and emphasise the considerable difference in the value of CS, we raise the CR of that

3

particular module by an exponent of ‘e’. This would not have been attained if we had just multiplied CR with the remaining product (i.e. KLOC*CC*CP/KLOCavg).

The next step is to attach percentage stringency to

each priority weight. By percentage stringency, we mean the maximum allowable deviation from the optimum. For example, a percentage stringency of 25% would mean that the actual value may exceed the optimal. by at most 25%. The maximum allowable deviation varies for time and cost and is also different for different organizations, based on their organizational policies. Following table shows the maximum allowable deviation assumed in our model for each priority weight.

Table 2: Priority Types Associated With Priority

Weights Priority Weight Percentage

Stringency for time

Percentage Stringency

for cost 1 25% 30%

1+2 20% 25% 1+2+3 15% 15%

1+2+3+4 10% 15% 1+2+3+4+5 5% 10%

Every component will have an optimal testing time T*(calculated from [5]), and the actual testing time Ta (i.e. the time taken to test the component). Similarly, there is an expected testing cost C0, and the actual cost of testing Ca associated with every component. We now introduce three variables into our system: α, β and δ. α is the deviation from optimal testing time, given by the equation [10]

α= (Ta – T*)/T* (4) β is the deviation from expected cost of testing. It is

calculated by the equation [9] β= (Ca – Co)/Co (5)

δ, the Limiting Factor, is the sum of deviations from optimal time and deviation from expected cost. It is represented by the equation

δ= kα + β.for some k>0 (6) k= ratio of priority of time over priority of cost.

This factor ‘k’ is to be defined uniquely for every project by the concerned people in the organisation. For instance, a cost intensive project, where time is of lesser importance than the cost, which is critical would have k<1. Similarly a time intensive project would have a value of k>1.

Based on the above equations, and the percentage stringency assigned to priority weights, we calculate the δmax i.e. the maximum allowable deviation (in terms of optimal testing time and expected cost) for each priority weight cumulatively. Note here that by cumulative, we mean that we take For example; take components with priority weight 1. In this case the maximum allowable stringency (i.e. deviation from optimal) is 25% for the optimal testing time, and 30% for the expected testing cost. Say, T* has a value 100 units and Co has a value

500 units. Ta and Ca would be 125 units and 650 units respectively (since we allow at most 25% deviation for time and 30% deviation for cost. We calculate the corresponding αmax and βmax values from the equations formulated above.

αmax= (125 – 100)/100 βmax= (650 – 500)/500

Assume k=0.5 i.e. priority of cost is 2 times the priority of time

δmax= (0.5)0.25+ 0.3= 0.425

Similarly, we can calculate the δmax for each cumulative priority weight. 5. Component Rank

In this paper we have suggested a formula to calculate the complexity of a module on the basis of factors like relative code complexity, relative code length and the dependence of other modules on the particular module and vice versa called Component Rank. This technique is similar to the Page Rank [14] technique used by Google for determining the relative importance of pages in a set of documents. Assume a small universe of four code segments: A, B, C and D. The initial approximation of Component Rank would be evenly divided between these four segments. Hence, each segment would begin with an estimated Component Rank of 0.25. If segments B, C, and D each only link to A, they would each confer 0.25 Component Rank to A. All Component Rank CR ( ) in this simplistic system would thus gather to A because all segments would be pointing to A. CR(A) =CR(B) + CR(C) + CR(D)

But then suppose segment B also has a link to segment C, and segment D has links to all three segments. The value of the link-votes is divided among all the outbound links on a segment. Thus, segment B gives a vote worth 0.125 to segment A and a vote worth 0.125 to segment C. Only one third of D's Component Rank is counted for A's Component Rank (approximately 0.083). CR (A) =CR (B)/2 + CR(C)/1 + CR (D)/3

In other words, the Component Rank conferred by an outbound link L ( ) is equal to the segment's own Component Rank score divided by the normalized number of outbound links (it is assumed that each segment calls the other only on, in case of loops it should be divided appropriately).

In other words, the Component Rank conferred by an outbound link L( ) is equal to the segment's own Component Rank score divided by the normalized number of outbound links (it is assumed that each segment calls the other only once in case of loops it should be divided appropriately).

4

Fig.1: Component Rank 6. Proposed Optimal Testing Policy

The component testing will follow a fix trend, which is proposed by our model. According to our model, all the components with very high priority will be tested first; followed by all the components with lesser priority and so on.

Fig. 2: Illustration of Optimal Testing Policy

As suggested in the model proposed by Srivastava

and others in [9], after testing all the components with priority weight 1(implying very high priority), we calculate δ. If the δ value is found to be below the δmax value, we move on to test the components with priority immediately lower to the current components. After

testing components with priority weight 2, we calculate δ value by taking the sum of T*, Co, Ta and Ca values of the previously tested components plus the T*, Co, Ta and Ca values of the components being tested currently. If the δ value thus obtained is found to be less than the δmax, we move further to test the next components. An optimal testing strategy can be derived from the proposed Priority based model. If the calculated δ ≤ δ max, it implies that the deviation from optimal time and cost is within acceptable limits. For components with very high, high, or medium priority, If δ > δ max, the code must be refined before testing any further. But, if δ > δ max for components with less or very less priority, we release the software instead of modifying or improvising the code any further, because it is not possible to test the software exhaustively. 7. An Example

Consider the example where the priority weights of α and β are given by the table2 and their percentage stringency associated with their priority weights are given by table3.

Table 3: Percentage Stringency Associated

With Priority Weights Priority Weight

αmax βmax

1 0.25 0.30 2 0.20 0.25 3 0.15 0.15 4 0.10 0.15 5 0.05 0.10

The value of δmax is given by the formula

δmax = kαmax + βmax Now we test the proposed model for different values

of k to verify if it’s possible to prioritize time over cost or cost over time. So the value of δmax is given in the following table 4.

Table 4: Calculation of max δmax Calculations… When K=

1 5 0.5 0.550 1.550 0.425 0.450 1.250 0.350 0.300 0.900 0.225 0.250 0.650 0.200 0.150 0.350 0.125

In the following table 5, the values of α and β are

calculated using the values of T*, C0, Ta, Ca from the given formulas in (4) and (5). The values of T*, C0, Ta, Ca that have been considered here, have been taken from a live project handled by a software organization.

5

Table 5: Sample Values of T*, C0, Ta and Ca

T* C0 Ta Ca α β 100 500 120 640 0.200 0.280 175 900 190 1100 0.086 0.222 225 1200 250 1330 0.111 0.108 250 1400 265 1750 0.060 0.250 265 1500 275 1600 0.038 0.067

Table 6: Calculation of Calculations of δ at K=

1 5 0.5 0.480 1.280 0.380 0.308 0.651 0.265 0.219 0.664 0.164 0.310 0.550 0.280 0.104 0.255 0.086

In table 6 shown above, values of δ for different

values of k are calculated. If the consumer has a higher time priority over cost k>1, if he/she has same cost and time priorities, then k=1 and if the priority of cost is greater than that of time, then k<1. We then calculate the value of δ and when δ> δmax the product is ready to be released. As we can see, for k = 1 the in the fourth row the value of δ= 0.310 is more than that of δ max = 0.250. As the higher priority modules have already been tested, and since it’s a lower priority module where δ exceeds δ max, we can release the software, in support of the proposed model.

Fig3: Variation in values for different values of k

The graph plotted here shows a variation in the values α, β, δ, δmax of at k=1. In the graph, the point where δ meets δmax the first time, at that point δ>δmax, the product is considered for release depending upon the priority of the module being tested. If the priority of the module being tested has a very less priority, then the product is released, while else, it is sent back for further testing when there is a scope for improvisation.

8. Conclusion

This proposed Deterministic Model for Test Case Prioritization using Non Homogeneous Poisson Process optimizes the time and cost of testing the entire software and gives a choice to the consumer to prioritize time over cost or vice versa. This approach optimizes the testing cost and time for individual components. By giving priority weights to the software modules in the model, its made sure that critical errors do not pass undetected during testing. Thus, the exact time when the software can be released can be predicted using the above proposed model. 9. References [1] Roger S Pressman, “Software Engineering: A Practitioner’s Approach” 6th Edition 2006. [2] Aditya P. Mathur, “Foundations of Software Testing. First edition 2008, perason education.” [3] McDaid Kevin and Wilson Simon P. (2001), “Deciding How Long to Test Software,” The Statistician, Royal Statistical Society, Part 2, 50, pp. 117-134. [4] Dalal S.R., and Mallows C.L., “When Should One Stop Testing Software?” Journal of the American Statistical Association, Vol. 83, No. 403 (Sep., 1988), pp. 872-879 [5] Musa, J.D., and Ackerman, A.F. (May 1989), “Quantifying software validation: when to stop testing?” IEEE Software, Vol. 6, Issue 3, pp. 19-27. [6] Ehrlich, W., Prasanna, B., Stampfel, J., and Wu, J. (March 1993), “Determining the cost of a stop-test decision [software reliability],” IEEE Software, Vol. 10, Issue 2, pp. 33-42. [7] Goel, A.L., and Okumoto, K. (1981), “When to stop testing and start using software?” Proc. of ACM, pp. 131-137. [8] Goel, A.L., and Okumoto, K. (1980), A Time Dependent Error Detection Rate Model for Software Performance Assessment with Applications, Proc. National Computer Conference, RADC-TR-80-179, May 1980. [9] Goel, A.L., and Okumoto, K. (1979), “A Time Dependent Error Detection Rate Model for Software Reliability and Other Performance Measures,” IEEE Transactions on Reliability, Vol. R-28, No. 3, August 1979, pp. 206-211 [10] Praveen Ranjan Srivastava et al. Non Homogenous Poisson Process Based Cumulative Priority Model for Determining Optimal Software Testing Period, ACM SIGSOFT Software Engineering Notes, Vol 33, No.2, March 2008. [11] Okumoto, K., and Goel, A.L. (1980), “Optimal Release Time of Software Systems Based on Reliability and Cost Criteria,” Journal of Science and Software, Vol. 1, No. 4, 1980.

6

[12] Praveen Ranjan Srivastava "Test case prioritization" Journal of Theoretical and Applied Information Technology, vol 4 no 2, 2008, pp 178-181. [13] Praveen Ranjan Srivastava et al. Test Case Prioritization Based on Requirements and Risk Factors , ACM SIGSOFT Software Engineering Notes, Vol 33, No.4, July 2008. [14] “Page rank Explained. Google's Page Rank and how to make the most of it.” http://www.webworkshop.net/pagerank.html