improving defect yield - a three step approach

Improving defect yield - A three step approachPowered by HBT

T AshokFounder & CEOSTAG Software Private Limited

in.linkedin.com/in/AshokSTAG Ash_Thiru

© 2013 STAG Software Private Limited. All rights reserved.

Defect Yield

2

Defect yield is the #defects (of good quality) uncovered by the test cases

Yield = Outcome/Effort

Defect yield = #Defects / # TestCases


Expectation & Challenges

3

ExpectationIncreasing the yield - a natural expectation.

ChallengesDefect yield drops/is-low : - when system matures - when system of testing is “weak” - when testing is transitioned to a different team


Why Lower Yield ?

4

Are we wasting effort by testing stable parts?

Are we missing doing something else?


The 3-step approach to yield improvement

5

Potency assessment1

Potential defect type re-targeting2

Potency improvement3

What types of defects are being targeted by the test cases?Are the test cases complete/adequate?

Are there any other types of defects to be targeted?

Add new test cases to cover the new types of defectsEnhance existing test cases to ensure that they are complete


Hypothesis Based Testing - HBT

6

System Under Test

Cleanliness Criteria

Potential Defect Types

Test CasesRequirementstraceability“what to test”

Faulttraceability“test for what”

should satisfy impeded by

© 2013 STAG Software Private Limited. All rights reserved. 7

Quality Levels & PDTs - Powered by HBT

Input cleanliness

Input interface cleanliness

Structural integrity

Behaviour correctness

Environment cleanliness

Attributes met

Flow correctness

Clean Deployment

End user value

L1

L2

L3

L4

L5

L6

L7

L8

L9

That inputs are handled wellPDTs related to Input data correctness

That the functional behaviour is correctPDTs related to Functionality

That the internal structure is robustPDTs related to Structural aspects

That the user interface is cleanPDTs related to UI

That end-to-end flows work correctlyPDTs related to Flow behaviour, Interactions

That it does not mess up the environmentPDTs related to Resource leaks, Compatibility...

That the stated attributes are metPDTs related to Performance, Load, Volume......

That it deploys well in the real environmentPDTs related to Compatibility, Migration

That user expectations are metPDTs related to User flows, User experience


Potency Framework (For Assessment & Re-target)

8

QL9 End user value End user value testPDT 57-59

QL8 Clean deployment Installation test PDT 54-55

Migration testPDT 56Migration test

QL7 Attributes met LSPS testPDT 44-51

Reliability testPDT 52Reliability testPDT 52

Security testPDT 53

QL6 Environment cleanliness Good citizen testPDT 39-41

Compatibility testPDT 42-43Compatibility test

QL5 Flow correctness Interaction testPDT 35-38

QL4 Behaviour correctness Functionality testPDT 24-31

Access control testPDT 32-34Access control test

QL3 Structural integrity Structural testPDT 14-23

QL2 Input interface cleanliness API validation testPDT 5-7

GUI validation testPDT 8-13GUI validation test

QL1 Input cleanliness Input validation testPDT 1-4Input validation test

QL - Quality LevelPDT - Potential Defect Type

Pow

ered

by

HB

T


An illustration : A component of a large system

9

Component = C1Component = C1Component = C1Component = C1Component = C1Component = C1Component = C1Component = C1Component = C1Component = C1Component = C1Component = C1BeforeBeforeBeforeBeforeBefore ADDITIONS (After)ADDITIONS (After)ADDITIONS (After)ADDITIONS (After)ADDITIONS (After)

#PDTs

#TCs#TCs#TCs #Defects '+PDTs '+TCs'+TCs'+TCs #Defects

Total '+ve '-ve Total '+ve '-veL8 0 0L7 1 1 1 0 1 2 1 1L6 0 0L5 0 0L4 7 1116 976 140 8 7 45 26 19 17L3 1 135 130 5 0L2 2 29 26 3 0L1 3 23 9 14 0Total 14 1304 1142 162 8 8 47 27 20 17

Increase byIncrease byIncrease byIncrease byIncrease by 57% 4% 2% 12% 213%

TC YieldTC Yield 0.01 Defects/TCDefects/TC TC YieldTC Yield 0.36 Defects/TCDefects/TC

Potency assessment PDT re-targeting21

3 Potency improvement


An illustration - The entire system

10

Potency assessment PDT re-targeting21

3 Potency improvement

Customer X : SummaryCustomer X : SummaryCustomer X : SummaryCustomer X : SummaryCustomer X : SummaryCustomer X : SummaryCustomer X : SummaryCustomer X : SummaryCustomer X : SummaryCustomer X : SummaryCustomer X : SummaryCustomer X : Summary

BeforeBeforeBeforeBeforeBefore ADDITIONS (After)ADDITIONS (After)ADDITIONS (After)ADDITIONS (After)ADDITIONS (After)

#PDTs #TCs#TCs#TCs #Defects '+PDTs '+TCs'+TCs'+TCs #Defects

Total '+ve '-ve Total '+ve '-ve

C1 14 1304 1142 162 8 8 47 27 20 17

C2 20 543 353 190 6 8 176 114 62 7

C2 6 741 494 247 19 7 17 12 5 0

Total 40 2588 1989 599 33 23 240 153 87 24

Increase byIncrease byIncrease byIncrease byIncrease by 58% 9% 8% 15% 73%

TC YieldTC Yield 0.01 Defects/TCDefects/TC TC YieldTC Yield 0.10 Defects/TCDefects/TC

10x yield improvement


HBT OverviewSIX staged purposeful activities, powered by EIGHT disciplines of thinking

11

D1D2

D4

D3

D5D6

D7

D8

Core Concepts

S1

S2

S3S4

S5

S6SIX Stages of DOINGS1: Understand expectationsS2: Understand contextS3: Formulate hypothesisS4: Devise proofS5: Tooling supportS6: Assess & Analyse

EIGHT Disciplines of ThinkingD1: Business value understandingD2: Defect hypothesisD3: Strategy & PlanningD4: Test designD5: ToolingD6: VisibilityD7: Execution & ReportingD8: Analysis & Management

Uses 32 Core ConceptsFor Problem solving - Techniques, Principles, Guideline

Click here to know more about HBT.http://stagsoftware.com/blog?p=570

http://stagsoftware.com/blog?p=570

http://stagsoftware.com/blog?p=570

http://slidesha.re/qBMNiy

http://slidesha.re/qBMNiy


HBT is the intellectual property of STAG Software Private Limited.STEMTM is the trademark of STAG Software Private Limited.

www.stagsoftware.com

@stagsoft

blog.stagsoftware.com

Connect with us...

Improving defect yield - A three step approachPowered by HBT

Potency assessment1

Potential defect type re-targeting2

Potency improvement3

Thank you.

http://www.stagsoftware.com

http://www.stagsoftware.com

improving defect yield - a three step approach

Technology