Reporting Software Quality Mark Fewster, Grove Consultants

Content

measuring software quality defect-based reporting risk-based reporting benefit-based reporting delivery-based reporting summary

2.2

Measuring software quality

what are we doing? assigning a number to something accuracy – guess / ruler / precision instrument

in testing we measure number and severity of defects found extent (coverage) of the software/system tested percentage of tests run (each cycle) number of test – debug – retest cycles time and effort taken non-functional attributes (e.g. performance, usability)

some measurements are more exact / objective than others

Measuring – lessons for testing

software quality has many aspects functionality, non-functional characteristics, defects

one measure doesn’t fit all!

choose the quality of your measurement gold / silver / bronze testing

reporting we report how bad – why not how good?

tests passed and predict future defect levels

Content

measuring software quality defect-based reporting risk-based reporting benefit-based reporting delivery-based reporting summary

2.5

How good is the software?

2.6

But this doesn’t tell us anything about the quality of the software now!

What about the unknown bugs?

We’ve tested it and we found 40 bugs.

38 bugs have been fixed. There are 2 bugs outstanding.

What about the tests that passed?

Known knowns and unknown unknowns

2.7

Known knowns: Unknown unknowns: bugs found bugs not found bugs shown not to exist (non-bugs) number of bugs not found

Knowing the unknowns

2.8

So which is it?

How can we tell?

Predict by using DDP!

Quote

2.9

“There is no point in asking how good the software is

until we know how good the testing was.”

Defects found in testing

Defects found in testing

Start Release

Not found - yet

How effective are we at finding faults?

Defects found after testing

or

Benchmark point

Defects found afterwards

“Really good!” “OK”

“Not bad” “Could be better”

“Poor” “Terrible!”

50 100 Defects found

after testing: Total defects

found:

Release

50 0

50 45 40 35 30 25 20 15 10 5 0

Defects Found

Time

62 12 62 81

69 19 69 72

74 24 74 68

77 27 77 65

85 35 85 59

87 37 87 57

88 88 38 57

10 Defects found

in testing: 42 50

Effectiveness at finding defects

100% 90% 80% 70% 60% 50% 40% 30% 20% 10%

0%

DDP

50 DDP = % =

Prediction of remaining bugs

20

10

66%

20

20

50%

20

5

80% Bugs found so far

DDP

Predicted bugs not found yet

Known knowns and known unknowns

2.13

Known knowns: bugs that have been found

Known unknowns: estimated bugs that have not been found

DDP 80%

Quote

2.14

Watts Humphrey, MIT

“If you want quality software out of testing,

you have to put quality software in to testing.”

Quality of testing

coverage has testing ‘covered’ all the important bits?

the functionality the non-function issues (performance, usability, etc.)

thoroughness has testing dug deep enough in each area?

typical data extreme data combinations

prioritisation has testing focused most on highest priorities?

2.15

Some stakeholders are not interested in bugs

bugs fixed are an historical detail they have been dealt with not about to influence what happens next

outstanding bugs are a ‘technical detail’ not always meaningful impact difficult to assess

consider car service

predicted bugs are hard to understand not meaningful

“something will go wrong, we don’t know what and we don’t know when”

impact impossible to assess 2.16

Content

measuring software quality defect-based reporting risk-based reporting benefit-based reporting delivery-based reporting summary

2.17

Risk-based testing

(product) risk is the possibility of a product failure describes possible consequences of product failures can express the impact on the business

identify product risks what could go wrong?

assess identified product risks how likely is it that the failure will happen? what would the impact be?

impact on end-user/business

use testing to mitigate highest level risks more thorough testing appropriate type of testing

2.18

Generic factors:

frequency of use business loss potential financial, ecological or social losses or liability civil or criminal legal sanctions safety concerns

fines, loss of license lack of reasonable workarounds visibility of the feature visibility of failure leading to negative publicity and damage to image loss of customers

1.19

Factors influencing business risk

¥

€

£ $

¥ €

£

$ ¥ €

£ $

Risks and tests

2.20

maintain traceability between risks and tests

when all tests for a risk have passed risk is mitigated

Risk 1

Risk 2

Risk 3

Test condition

Test condition

Test condition Test

condition Test

condition Test

condition

Test case

not run

passed failed

not run

passed failed

not run

passed failed

not run

passed failed

Test case

Test case

Test case

Risk 1: mitigated

Risk 2: outstanding

Risk 3: outstanding not run

passed

failed

passed

Risk-based reporting

2.21

Progress through the test plan

today end date

residual risks of releasing

TODAY

start

Source: “Risk Based E-Business Testing”, Paul Gerrard & Neil Thompson

all risks ‘open’ at the start

Content

measuring software quality defect-based reporting risk-based reporting benefit-based reporting delivery-based reporting summary

2.22

Benefit based test reporting

2.23

Open

Closed

Ris

ks

Open

Open

Closed

Closed

Open

Ben

efit

Ben

efit

Ben

efit

Ben

efit

Ben

efit

Ben

efit

Ben

efit

Ben

efit

Ben

efit

Benefits available for release

Ben

efit

Ben

efit

Focusing on certain risks will deliver different benefits

Closed

Source: “Risk Based E-Business Testing”, Paul Gerrard & Neil Thompson

Content

measuring software quality defect-based reporting risk-based reporting benefit-based reporting delivery-based reporting summary

2.24

Agile approach

a user story describes functionality that will be valuable to a user or purchaser three parts (usually hand-written):

description notes of conversations acceptance tests

3.25 Front

Users can view list of account transactions for a selected period. Richard said to show transactions up to 12 months old.

Back

Users can view list of account transactions for a selected period. Richard said to show transactions up to 12 months old.

• Try with 1000’s of transactions. • Try with no transactions. • Try a 1-day period.

User story testing

each user story includes acceptance tests

completion criteria non-functional criteria an indication of size and complexity: story points

a small number of user stories are implemented within a single iteration

designed, developed, tested and demonstrated total number of story points = velocity

3.26

Content

measuring software quality defect-based reporting risk-based reporting benefit-based reporting delivery-based reporting summary

2.27

Reporting software quality: summary

defect-based reporting is a negative view, can overlook what works is often a technical view

not always meaningful to the business

risk-based reporting assessment based on impact on business reporting mitigated risks more positive

benefit-based reporting focus on what can be delivered successfully

delivery-based reporting agile focused on user stories – what the user can do incremental approach, proof by demonstration

3.28

A final thought

3.29

“The accuracy of the reporting is dependent on the

quality of the testing.”