a survey of unit testing practices

1

A Survey of Unit Testing Practices

Prof. Per Runeson – Lund University

Per Beremark – SPIN-syd

2

What is Unit Testing?

“testing of individual software or hardware units or groups of related units” [IEEE610.12]

“Unit testing tests individual software components or a collection of components. Testers define the input domain for the units in question and ignore the rest of the system. Unit testing sometimes requires the construction of throwaway driver code and stubs and is often performed in a debugger” [Whittaker 00]

“a test, executed by the developer in a laboratory environment, that should demonstrate that the program meets the requirements set in the design specification” [Koomen and Pol 00]

3

What Does It Mean In Practice?

1. What do people refer to as unit testing?

2. What are the strengths regarding unit testing in industry?

3. Which are the problems regarding unit testing?

4

Key Findings on Unit Testing

• What?– Technical!– Specified or not?

• How?– Structure-based– Not formally

• Who?– Developer (team)– Not test or QA

• When?– Each

build/day/week– Takes seconds to

hours to run

• Why?– Assuring

functionality

5

Participants:SPIN-syd of Southern Sweden

• 19 companies– Telecom– Automation– Case tools– Information

systems– Banking– Health care– Consulting– Transportation

• Company size– 1-1000

developers

• 17 participants in focus group

• 15 in questionnaire

6

Survey Procedures

Unit testing is…

Unit testing-What-Strengths-Problems

Analysis-What?-How?-Where?-Who?

Q’s

Note

1. Individual reflection

2. Focus group discussion

Note

3. Document and structure

4. Analysis

5. Survey preparation

6. Survey conduct

7. Survey analysis

Q&A’sQ&A’s

Q&A’s

Note

7

3rd party verification

Unit test – smallest specified unit

Test first

Training

Focus Group Meeting

8

Analysis Framework

• What?• How?• Where?• Who?• When?• Why?

Unit test

Unit test

Unit test

Unit test

Unit test

Unit test

Unit test

Unit test

9

Questionnaire

10

Analysis

0

2

4

6

8

10

12

14

1. S

mall

est u

nit

2. T

echnica

l

3. F

unctio

ns

4. S

epar

ate

5. S

truct

ural

6. C

over

age

7. A

utom

ated

exec

ution

8. A

utom

ated

follo

w-up

9. S

pecif

ied in

text

10. S

pecifie

d in

test

code

11. E

xecu

ted

by d

evelop

ers

12. D

evelo

pers/t

ester

s

13. T

est d

epartm

ent

14. Q

uality

dep

artm

ent

15. F

or e

ach

build

16. M

any t

imes

eac

h da

y

17. A

t lea

st da

ily

18. A

t lea

st wee

kly

19. S

econds

20. M

inutes

21. H

ours

22. U

nit fu

nction

s

23. A

ccept

s unit

s

24. S

pecify

unit

25. Im

prov

e qua

lity

26. C

usto

mer

req's

Fully agree

Agree

Neutral

Disagree

Totally disagree

N/A

What? Time?When?Who decides?Who conducts?How? Why?

11

What?

• Definition– Test of smallest unit or units

• Strength– Unit identification– Test of surrounding modules

• Weaknesses– GUI test– Unit identification– Test scripts and harness maintenance– Data structures

12

How?

• Definition– Structure-based– Preferably automated

• Strength– Test framework

• Weaknesses– Documentation– Framework tailoring– Test selection– Test metrics

13

Who?

• Definition– By developer

• Strength– Independent test– Competence network

• Weaknesses– Competency– Independence– Introduction strategy

14

When?

• Definition– Quick feedback

• Strength– Continuous regression test

• Weaknesses– Stopping criteria

15

Why?

• Definition– Ensure functionality

• Strength– External requirement (safety)– Agile methods

• Weaknesses– Cost vs. value

16

Further Use of the Survey

• Clarify unit test definition in your company

• Lightweight assessment of unit testing practices– Benchmark– Improve

17

Read more… … in IEEE Software July/August 2006

18

http://serg.telecom.lth.se

19

Definitions

0

2

4

6

8

10

12

14

1. S

mall

est u

nit

2. T

echnica

l

3. F

unctio

ns

4. S

epar

ate

5. S

truct

ural

6. C

over

age

7. A

utom

ated

exec

ution

8. A

utom

ated

follo

w-up

9. S

pecif

ied in

text

10. S

pecifie

d in

test

code

11. E

xecu

ted

by d

evelop

ers

12. D

evelo

pers/t

ester

s

13. T

est d

epartm

ent

14. Q

uality

dep

artm

ent

15. F

or e

ach

build

16. M

any t

imes

eac

h da

y

17. A

t lea

st da

ily

18. A

t lea

st wee

kly

19. S

econds

20. M

inutes

21. H

ours

22. U

nit fu

nction

s

23. A

ccept

s unit

s

24. S

pecify

unit

25. Im

prov

e qua

lity

26. C

usto

mer

req's

Fully agree

Agree

Neutral

Disagree

Totally disagree

N/A

What? Time?When?Who decides?Who conducts?How? Why?

20

Strengths and weaknesses

0

2

4

6

8

10

12

1. Id

entify

units

2. G

UI com

ponen

ts

3. D

ata

depe

nden

cy

4. E

xtern

al co

de

5. R

eal-t

ime

6. T

est ca

se se

lectio

n

7. M

aintena

nce

8. S

pecif

y test

case

s

9. A

utom

ate

testi

ng

10. B

uild/ta

ilor f

ram

ewor

ks

11. B

uild sy

stem

s

12. C

M sy

stem

s

13. T

rouble

repo

rting

14. D

ocum

enta

tion

15. S

caffo

lding

16. C

over

age

17. D

evelo

pers e

xecu

te a

ll

18. E

xter

nal u

nits

19. P

riorit

y and

status

20. C

ompe

tenc

e and

skills

21. F

reque

ncy

22. C

riter

ia

23. M

otiva

tion

24. G

ain a

waren

ess

Very goodGoodNeutralBadVery badN/A

What? When?How? Why?Who?