balamurali l senior sqa manager [email protected] diana ambrose senior lead sqa...

104/19/23

ISQT’s STEP-AUTO 2011: Conference on ERP TestingApril 19, 2023 2

Software Systems – A Part of Daily LifeBanking

applications

Home appliances

Educational applications

Cars and other vehicles

Communicati-on Media

Software is now

omnipresent

ISQT’s STEP-AUTO 2011: Conference on ERP Testing

• Reduction of errors• Increasing confidence of end users• Ensuring safe operations• Contributes to quality• Reduces cost of defect fixing

3

Role of Testing

04/19/23


Testing: Perspective shift…

• Establish confidence that a program does what it is supposed to do (Hetzel, 1973)



• The process of executing a program or system with the intent of finding errors (Myers, 1979)



• Verifying that a system satisfies its specified requirements and identifying the differences between expected and actual results

ISQT’s STEP-AUTO 2011: Conference on ERP Testing 7

The Lacuna

04/19/23

SQT’s STEP-AUTO 2011: Conference on ERP Testing04/19/23 8

Where we stand now…

You can manage, what you can measureYou can manage, what you can measure

You can measure, what you can defineYou can measure, what you can define

You can define, what you understand.You can define, what you understand.


bug prediction and tracking

904/19/23


Introduction

• Uses baseline data of the organization and size of the current project

• Predicts the expected number of bugs and the testing schedule for the current project

• Cumulative bug trend curve resembles the curve of a first order response system

• The actual cumulative bug trend is tracked against the estimated

• Any notable deviation from the bug prediction curve, analysis will be triggered and corrective actions taken accordingly.

1004/19/23


Cumulative Bug Trend Curve

• A curve with:– The days of

testing marked on X-Axis

– The cumulative number of bugs caught on Y-Axis

• The slope of the curve gives the rate of bug capture

050100150200250300350400

24-Jan 29-Jan 3-Feb 8-Feb 13-Feb 18-Feb 23-Feb 28-Feb 5-Mar 10-MarC

umul

ativ

e B

ugs

Day

CumulativeBug-TestDay Chart

04/19/23


• Monitoring– Bug Prediction Curve– Weekly Status Report– % Schedule Completed– Resource Utilization

Process In Testing - Monitoring

Schedule

ODB

ProjectTeam

No. Testing

Days

DefectDensity

Size

Defect Prevention measures

Expected Defects & Schedule

1204/19/23

CumulativeBug-TestDay Chart

0

100

200

300

400

24-Jan 3-Feb 13-Feb 23-Feb 5-Mar 15-Mar

Day

Cu

mu

lati

ve B

ug

s

Est vs Act Total Bugs Curve

0

50

100

150

200

0 2 4 6 8 10

No. Days of Testing

Cu

mu

lati

ve N

o. B

ug

s

EstDefectActDefectsULEst

LLEst


Bug Prediction Curve

Number of days

Cum

ulati

ve n

umbe

r of b

ugs

Helps in tracking the test execution

04/19/23


Bug Prediction Curve

0

50

100

150

200

0 2 4 6 8 10

Cumu

lative

No.

Bugs

No. Days of Testing

Bug Trend Curve

Est Defect

ULEst

LLEst

04/19/23


Possible Outcomes

• The actual curve - within expected trend• The actual curve - outside expected trend– The number of bugs in the initial quarters is much less

than estimated– The number of bugs in the initial quarters is much more

than estimated

04/19/23


The Scenario Diagram

Situations

Bugs in initial stage :Less than what is

estimated

Bugs in initial stage :More than what is

estimated

Stable Product InadequateTesting

UnstableProduct

04/19/23


Inadequate Testing

0

50

100

150

200

0 2 4 6 8 10

Cumu

lative

No.

Bugs

No. Days of Testing


Est Defect

Act Defects

ULEst

LLEst

04/19/23


Unstable Product

0

50

100

150

200

0 5 10

Cumu

lative

No.

Bugs

No. Days of Testing


Est Defect

Act Defects

ULEst

LLEst

04/19/23


Challenges in ERP Testing

• Wide range of stakeholders that use the system• Impact of changes across various modules and higher

risk of issues to be found at integration• Inherent slippages that is accumulated over various

phases of the development lifecycle• Tracking and Managing multiple test iterations in parallel• High pressure to ensure delivery dates as well as the

quality of delivery

1904/19/23


In the ERP context

• Bug trend curve can be calculated for different iterations• Specification limits can be defined for each iterations• Actual bug trend curve can be plotted and monitored for each

iterations

2004/19/23


Bug Trend Curve for multiple iterations

2104/19/23


Test Status Indicator

2204/19/23


Test Status Indicator

• Root mean square of the variation between the estimated and actual bug trend curve

• The variation between the planned and actual value of the bugs can be calculated on a daily basis.

• The overall TSI values of the total projects in an organization can be plotted in a control chart for monitoring.

• The specification limits can be defined and from the control chart, the points which fall outside the limits can be analyzed by the top management.

2304/19/23


SPC for TSI

2404/19/23


SPC for TSI in different iterations

2504/19/23


Advantages

2604/19/23


Case Study

Project Language Plat form Start Date Cost of a Bug (Hrs/Bug)

% Testing Effort

Before implementation of the method

Project 1 VC Win NT 13-Mar-11 1.11 11%

Project 2 VC Win NT 21-Dec-10 1.28 37%

Project 3 C# Win NT 13-Mar-11 1.86 25%

After implementation of the method

Project 4 VC Win NT 16-May-11 0.68 13%

Project 5 VC Win NT 11-Jun-11 0.94 7%

Project 6 C# .net 29-Apr-11 0.63 7%

• Reduction in % effort for testing

• Implementation of Bug Prediction and Tracking method ensures quantitative monitoring and control

• Critical phases tracked in an efficiently and effectively

2704/19/23


Questions

2804/19/23


Thank you

2904/19/23

balamurali l senior sqa manager [email protected] diana ambrose senior lead sqa...

Documents

erp testingisqts stepauto

erp testingsqts stepauto

omnipresentsqts stepauto

actual resultssqts stepauto

erp testingbug prediction

testing schedule

testing monitoring

role of testing