m global software group 1 motorola internal use only better software quality at a lower cost:...

MGlobal Software Group1 Motorola Internal Use Only

Better Software Quality at a Lower Cost: Testing to Eliminate Software Black Holes

Isaac (Haim) Levendel, Director

Systems Engineering Technology Center,

Global Software Group


Central theme

• Process to accelerate software quality improvement by maximizing the impact of software testing, given:

– a limited testing budget– limited time

• We generally cannot afford full testing coverage

• How do we manage risk?


Traditional theory of software reliability:bugs are homogeneously distributed


Implications of the traditional theory of software reliability

1. Development data can be used to predict field readiness• that’s good!• But is the prediction correct?

2. Full coverage testing is necessary to improve the software• that’s bad!


Humans deal with complexity in an iterative way

• We use successive approximation methods• The lack of complete understanding of the correct

design space is the root cause of most software errors– These errors happen at all phases of the software

manufacturing process

• At every point in time, a SW black hole is– An incomplete/incorrect area of the software


Levendel’s Axiom

“Bugs Abhor Loneliness”

(They create software black holes)


Implications of the black hole theory of software reliability

1. Development data can be used to predict field readiness• that’s good!• But is the prediction correct?

2. Full coverage testing is necessary to improve the software• that’s bad!


Disintegration of software black holes

(4)

Software blackhole

(1) (2)

(3) (5)

Acceptable?


At the end of the process:the SW product variance has been reduced


The software crafting phases

Requirements

Design

Coding

Unit Design

Integration

“Make believe”

Requirements

Design

Coding

Unit Design

Integration

Reality


Requirements

Design

Software black holes are left behind anywhere in the crafting

process

Coding

Unit Design

Integration

Each phase leaves SW black holes


The black hole theory of software

Supporting data and experience


Software modules affected by changes for new features (Subsystem #1)

0102030405060708090

100

0 10 20 30 40 50 60 70 80 90 100

Percent of modules affected

Per

cen

t o

f ch

ang

es


Software modules affected by changes for new features (Subsystem #2)

0102030405060708090

100

0 10 20 30 40 50 60 70 80 90 100

Percent of modules affected

Per

cen

t o

f ch

ang

es


Number of new and modified lines of source code for each internal release

(total of 34 internal releases)

0100002000030000400005000060000700008000090000

100000

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33


Number of software modules changed per fix for each internal release

(total of 34 internal releases)

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33


Number of lines of software changed per fix for each internal release (total of 34 internal releases)

0

50

100

150

200

250

300

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33


Software field products will always have bugs

Cost PerBug Fix

Reproducibility (1/MTBF)

Time


Identifying black holes


Role of testing: help to identify black hole

(4)

Software blackhole

(1) (2)

(3) (5)

Acceptable?


Advancing the release time by repairing a software black hole in one step

B’

A = A black hole is found

B = Field delivery

Number of Tests t (or time elapsed)

Cu

mu

lati

ve N

o. o

f F

ailu

res

300

250

200

150

100

50

0

0 1,000 2,000 3,000 4,000 5,000

O

150

100

50

0

Climb to a “better” curve

= The black hole is fixedO’

A’


Testing to improve product quality

Home on SW Black Hole

Explore SW Black Hole

Repair SW Black Hole

(redesign SW module)

Stop

More SW

black holes? Yes

No

Find bug

Fix bug

Execute test

Retest fix

Stop

More

Tests? Yes

No


How do we position the initial testing on software black holes?

Desirable initial tests

Not worth testing


Information sources for guiding the testing toward potential software black holes

Proc. Phase

Info

Info type

Past releasesPlanning

of current release

Design/coding of current

release

Testing

of current release

Field data Laboratory problems SW change information SW planning problems

TIME


Correlation between trouble reports and fixes helps identify problem-prone

software modules

Trouble report 1

SW Module 1

Fix 1bFix 1a

SW Module 2

SW Module 3

SW Module 4

Trouble report 2

Fix 2a


Code changes and new code may affect problem-prone software modules

Code change request 1

SW Module 1

Change 1

SW Module 2

SW Module 3

SW Module 4

Code change request 2

Change 2


White box testing leads to higher fail rate

Trouble report 1

SW Module 1

Fix 1bFix 1a

SW Module 2

SW Module 3

SW Module 4

Trouble report 2

Fix 2a

Test x

fails

Test y

passesTest z

fails


Test program alerts

• Test programs cannot be fully designed one year ahead of time!!!– because one does not know in advance all the places where

the functionality is going to break

• They must be adaptive!!!– Need to combine upfront planning with adaptive execution

• This includes regression tests


Practical Testing Question

Black Box Testing

or

White Box Testing?


Practical Testing Question

Black Box Testing

or

White Box Testing?

Both!! Why?

•Black box testing reflects requirements•White box testing reflects software black holes


Who would want to test like that?


4 weeks of testing data

Testers withoutknowledge aboutSW black holes

1 55 48

2 50 46

3 49 45

4 27 25

5 26 24

6 21 21

. . .

37 1 1. . .

44 1 045 0 0. . .

51 0 0

TesterProblemsopened

Actualproblems

Testers with knowledge about SW black holes

25 12 5...


Test program architecture

&

Test program metrics


Managing complexity• Complexity can be managed by decomposing large

systems into a tree-like hierarchy of functions• The depth and levels of the decomposition are arbitrary

Functional Area

Functional Package Functional Package

Functional Subset Functional Subset

Functional Functional Functional Component Component Component


How a test suite traverses functional components

Test 1 Test 2

Component 1

Component 2 Component 3


Component 6


Two key issues with components

1. The SW architecture is a good starting point for a decomposition into functional components

2. It is essential to characterize the mapping between test suites and functional components


Implementing test programs: the combinatorial explosion problem

Functional Area

Functional Package

Functional Subset

FunctionalComponent

Functional Package

Functional Subset

FunctionalComponent

FunctionalComponent

Underlying

Functional

structure

Unmanageable

size of

test program+ Feature

interactions


Filtering test program scope to reduce cost

Functional Area

Functional Package

Functional Subset

FunctionalComponent

Functional Package

Functional Subset

FunctionalComponent

FunctionalComponent

Customer

rollout plan

Suspected

vulnerabilities

Underlying

Functional

structure

Manageable

Test

program


Collectfailures on tests ( ) and components ( )

Test 1 Test 2

Component 1



Component 6


Fail rates for typical project’s components

0102030405060708090

100

1 2 3 4 5 6 7 8 9 10

Functional components

Fail Rates

Likely software black holes


Create time and space repetitions

FS2

FS3

FSn

FS1

FS2

FS3

FSn

FS1

FS2

FS3

FSn

FS1

FS2

FS3

FSn

FS1

Evaluation Evaluation Evaluation Evaluation Period 1 Period 2 Period 3 Period 4

Comparisons

Comparisons

Comparisons

Comparisons

Time repetition

Space repetition


A reporting mechanism to motivate repairs of software black holes

• It is more economical to climb the quality curve in steps

• The reporting should motivate these step functions• The solution:

– Drive the testing to identify software black holes

– Orient the test reporting to point at software black holes & recommend solutions


Log the learning

1. The exercise of human judgment is indispensable

2. Data per se has NO meaning


The key common sense question about testing in general

Common Sense is not so common - Voltaire (1694-1778)

• How does one measure testing productivity?– In number of tests executed per $?

or

– In number of test failed per $?


The key common sense question about testing in general

Common Sense is not so common - Voltaire (1694-1778)

• How does one measure testing productivity?– In number of tests executed per $?

or

– In number of test failed per $ !


(Dire) Conclusion

If one accepts the software black hole hypothesis, then:

the current reliability and availability technologies need to be rethought

m global software group 1 motorola internal use only better software quality at a lower cost:...

Documents