metrics for tracking system test

8/3/2019 Metrics for Tracking System Test

1/27

Click to edit Master subtitle style

5/6/12

Metrics for Tracking

System Test


2/27

5/6/12

Outline

Need to track system testing

Metrics required for system test

tracking Categories of execution metrics

Example metrics using real life

examples of project Bazooka andStinger


3/27

5/6/12

Basic Concepts

System testing occurs during the endof a projects life cycle. The threefacets at this stage are:

Developers need to know who well thesystem meets the explicit/implicitrequirements.

Delivery date is critical andunpredictable as time required to fixdefects is not known.

Customer is waiting anxiously for theroduct deliver .


4/27

5/6/12

Basic Concepts

Difference between statistics & in-process metrics.

Statistics Post project analysis to gather

experience and use it in future projects

In Process metrics in-process metrics are used to monitor

the progress of the project while there isstill an opportunity to steer the course

of the project


5/27

5/6/12

Basic Concepts

Revert criterion:

A condition if met, the current test cycleis stopped and the root cause of the

defects is found and fixed beforerestarting the test cycle.

Test Factory:

Automated tool to form a sort ofdatabase to store & categorize the test

cases and monitor the numbers.


6/27

5/6/12

Execution metricscategorization


7/27

5/6/12

Metrics for monitoring testexecution

Monitor execution of test cases

Large project, more test engineers,

more time taking Automated tools shall be used e.g.Test factory

In-process metrics are gathered fromthe database. These metrics are:

Test Case Escapes (TCE)

Planned versus Actual Execution (PAE)


8/27

5/6/12

Test case Escapes (TCE)

Test engineers need to design new test cases duringtesting.

A significant increase in the number of TCE impliesdeficiencies in the test design


9/27

5/6/12

PAE

Planned versus Actual Execution (PAE) Rate

Compare the actual number of test cases executedevery week with the planned number of test cases

This metric is useful in representing theproductivity of the test team inexecuting test cases

If the actual rate of execution falls farshort of the planned rate, managersmay have to take preventive measuresso that the time required for system

testing does not adversely affect the


10/27

5/6/12

Corrective actionsrequired

Testingexecutioncycleextended


11/27

5/6/12

Execution Status of Test (EST) Cases

Periodically monitor the number of test cases lyingin different states

Failed, Passed, Blocked, Invalidand Untested

Useful to further subdivide those numbers by testcategories


12/27

5/6/12


13/27

5/6/12

Revert criteria:

At any instant during the test cycle, ifthe cumulative failure rate reaches 20%,

the cycle will restart all over after thereported defects are claimed to havebeen fixed.

Execute most vulnerable test cases atthe beginning of the cycle.

Pass: 71.11% Fail:28.89%

Analysis:


14/27

5/6/12

Metrics for monitoringdefect reports

Function as Designed (FAD) Count

Defects which are not really true defectsbecause of a misunderstanding of the

system

The lower the FAD count, the higher thelevel of system understanding of the

test engineers


15/27

5/6/12

Irreproducible Defects (IRD) Count

One must be able to reproduce thecorresponding failure after a defect is

reported so that developers canunderstand the failure to be able to fixit.

Irreproducibility of a defect does notmean that the defect can be overlooked.


16/27

5/6/12

Defects Arrival Rate (DAR) Count

Defect sources:

System Testing group/ ST,

Software Developers/ SW,

Software Integration Group/ SIT,

Others (customer support, marketing etc.)

Percentage of defects reported by eachgroup on a weekly basis computed.

Most test cases


17/27

5/6/12

Defects foundby others is

lesser

Most test casesof vulnerable

areas areflushed


18/27

5/6/12

Defects Rejected Rate (DRR) Count

Once defects are reported, attempts aremade to correct them and the

corresponding test cases are rerun.

Some of the supposedly fixed defectsare not actually fixed.


19/27

5/6/12

Defects Closed Rate (DCR) Count

The count telling the number of defects fixed afterrerunning the test cases.

Higher DCR count, better product quality andtesting team productivity.


20/27

5/6/12

Rejected defectsreduce by the

end


21/27

5/6/12

Outstanding Defects (OD) Count

A defect that is not fixed and continues to exist.

Higher OD count, lower prevailing quality.


22/27

5/6/12

Critical

High Medium

Low

Total number of

defects reduced bythe end


23/27

5/6/12

Crash Defects (CD) Count

The number of defects causing the system to crashand resulting in unavailability & loss of data

Higher the CD count, lower the system stability


24/27

5/6/12

Crashes are scrutinized and theirroot causes are found and

eliminated


25/27

5/6/12

Arrival and Resolution of Defects (ARD)Count

Comparison of rate of defects found to the rate of

their resolution.

Used to estimate time required to fix all the defects.


26/27

5/6/12

E

sti

mate

d

Actual open defects are more thanestimated, so manager needs to

take corrective actions


27/27

5/6/12

Conclusion

At this stage, tracking & monitoringthe system tests is important to:

Measure testing progress

Reveal the system quality

Control progress

These metrics help the managementto take protective and correctiveactions to stay on the right track.

metrics for tracking system test

Documents