Software Testing Metrics

By David O’ Connor

What are Software Testing Metrics?

The definition of a Metric as stated by SoftwareTestingHelp. (2013) is a quantitative measure of the degree to which a system, system component, or process possesses a given attribute. Metrics is defined as Standards of Measurements.

SoftwareTestingHelp. (2013) states that software metrics are used to measure the overall quality of the software project. In simply terms metric is a unit used for describing an attribute, and is a scale for measurement.

Definition of Software Metrics given by Paul Goodman: - Software Metrics is a Measurement Based Technique which is applied to processes, products and services to supply engineering and management information and working on the information supplied to improve processes, products and services, if required.

Importance of Metrics

The importance of software testing metrics are stated below taken from Lokesh Gulechha. (2009):

Metrics is used to improve the quality and productivity of products and services and therefore will lead to achieving User Satisfaction.

Metrics makes it easy for software management to digest one number and drill down, if required.

Different Metric(s) trend act as monitor when the process is going out-of-control.

Metrics provides improvement for current process.

Defect Density Metric  Defect Density is defined as stated by STF. (2010) as the number of

confirmed defects which have been detected by the software tester in the particular software/component during a defined period of the development or operation divided by the overall size of the software application or component.

The ‘defects’ are:

Confirmed and agreed upon by the software testers and are not just reported.

Any defects which are dropped during the process are not counted.

The period can be defined as one of the following:

For a time over a duration for example such as the first month, the quarter, or even the year.

For each phase of the life cycle of the software.

For the overall life cycle of the software.

Defect Density Metric continued The size can be measured in either Function Points (FP) or by Source Lines

of Code.

The number of errors found in test cases versus test cases developed and executed

(Deflective Test cases/Total Test cases) * 100

Example: The total number of test cases developed is 1250, total test cases executed is 1150, total number of test cases passed is 1065, and total number of scripts failed is 190.

So Test case Deflect Density is: (190x100)/1150 = 16.5%

This 16.5% can therefore be called a Test Case Efficiency % which depends upon the total number of Test cases which were found to be deflective.

The Defect Density metric is used by testers for comparing the number of relative defects in the various aspects of software components so that these high-risk components can be easily identified and the main resources can therefore be focused towards on these deflects.

The Defect Density Metric should be gathered by the software tester because it gives a total percentage of the defective parts in the overall software component and therefore resources can therefore focus on the critical deflective parts of the component.

Requirement Volatility Metric The requirement volatility of software testing has a marked effect on the

overall deliverables of any software product development effort.

There the requirement volatility metric as stated by P.M.Venkatesh Babu. (2009) is in place to ensure that the requirements of the software procedure are normalized or defined properly while estimating the total number of requirements agreed versus the number of requirements that were changed.

The Requirement Volatility Metric formula is defined as the following:

(Number of requirements Added + Deleted + Modified) * 100 / No of original requirements

An example of this formula being used as demonstrated by P.M.Venkatesh Babu. (2009): SVN 1.3 release has 67 requirements initially, later 7 new requirements are added, 3 requirements are deleted from initial requirements and modified 11 requirements

Hence Requirement volatility is calculated as:

(7 + 3 + 11) X 100 / 67 = 31.34 %

P.M.Venkatesh Babu. (2009) stated that this result means that almost 1/3 of the requirements changed after the initial identification of requirements.

Requirement Volatility Metric continued

Requirements metrics such as the requirement volatility metric, traceability, size and completeness are used by software testers in order to accurately measure requirements engineering phase of software development so that this will therefore give the software tester a clear indication of how the product is performing under the various tests being carried out during the testing phase in order to ensure the end product is up to the specifications set out by the client and therefore this metric should be gathered to ensure requirements are up to standard.

Test Execution Productivity (TEP) The Test Execution Productivity metric as stated by Lokesh Gulechha.

(2009) gives the test cases execution productivity which on further analysis can give conclusive result.

Test Execution Productivity =

((Total No. of TC executed (Te)/Execution Efforts (hours))*8)) Execution(s)/Day

Where Te is calculated as,

Te = Base Test Case + ((T(0.33)*0.33)+(T(0.66)*0.66)+(T(1)*1))

Where,

Base Test Case = No. of TC executed at least once.

T (1) = No. of TC Retested with 71% to 100% of Total TC steps

T (0.66) = No. of TC Retested with 41% to 70% of Total TC steps

T (0.33) = No. of TC Retested with 1% to 40% of Total TC steps

Test Efficiency (TE) The Test Efficiency metric as stated by Lokesh Gulechha. (2009) is performed to

determine the overall efficiency of the software testing team in identifying the defective components of the software.

The Test Efficiency metric is also used to indicate the number of defects which were missed out during testing phase and which migrated to the next testing phase.

The Test Efficiency of the testing process is calculated by the following formula:

Test Efficiency = DT/(DT+DU)*100%

Where,

DT = The overall number of valid defects identified during the testing process.

DU = The number of valid defects which have been identified by test user after the release of software application. In other words, post-testing defect

The Test Efficiency metric should be gathered by the software tester because it gives a total percentage of the defective parts which were missed by the tester in the testing process for the software component and therefore this process will lead to better efficiency by the software tester when testing future software components.

References

Lokesh Gulechha. (2009). Software Testing Metrics. Software Testing Metrics. 1 (1), 7-8.

P.M.Venkatesh Babu. (2009). Software Testing Metrics. Available: http://www.slideshare.net/pmvenkateshbabu/testing-metrics. Last accessed 20th April 2015.

SoftwareTestingHelp. (2013). Important Software Test Metrics and Measurements – Explained with Examples and Graphs. Available: http://www.softwaretestinghelp.com/software-test-metrics-and-measurements/. Last accessed 09th Apr 2015.

STF. (2010). Defect Density Fundamentals. Available: http://softwaretestingfundamentals.com/defect-density/. Last accessed 18th April 2015.