experiences with tmmi facts and figures - romanian testing conference 2014

Experiences with TMMi: facts and figures

Jan Jaap CannegieterVice President SYSQA B.V.

© S

YS

QA

Alm

ere

2

Agenda

0

20

40

60

80

100

What is

A maturity modeland

an assessment model

© S

YS

QA

Alm

ere

4

Test Maturity Model integration

Goals and practices

Certified assessors

Method independent

Organization

independent

Open source

© S

YS

QA

Alm

ere

5

TMMi levels

1. Initial

2. Managed

Test Policy and StrategyTest PlanningTest Monitoring and ControlTest Design and ExecutionTest Environment

3. Defined

Test OrganizationTest Training ProgramTest Life Cycle and IntegrationNon-Functional TestingPeer Reviews

4. Management and Measurement

Test MeasurementSoftware Quality EvaluationAdvanced Peer Reviews

5. OptimizationDefect PreventionTest Process OptimizationQuality Control

© S

YS

QA

Alm

ere

6

Formal a

ssessment Informal assessment

Planning

Data management

Process attribute rating

Reporting and output

© S

YS

QA

Alm

ere

9

Assessment type: Quick Scan

Test manager 1

Tester 1

QA manager

Test environment administrator

Test manager 2

Tester 2

Project manager

You

8.30 Opening & TMMi

9.00 Rating PA’s level 2

12.00 Lunch

13.00 Rating PA’s

14.00 Conclusions

14.30 Determine improvement actions

16.00 Making action plan

16.30 Evaluation

(20.30 On the couch watching TV)

© S

YS

QA

Alm

ere

10

The tool

© S

YS

QA

Alm

ere

11

TimeStart

I I IQS QS QS QS QS

F

QS

Assessments and TMMi implementations

Level 2

© S

YS

QA

Alm

ere

12

Assessment types compared

0 100

Formal Informal Quick scan

Accuracy

Costs

100% Use at SYSQA

© S

YS

QA

Alm

ere

13

TMMi today

TMMi model

TAMAR

F i v e s e r v i c e p r o v i d e r s

i n f o u r c o u n t r i e s

22 assessors

1000 members

16 supporting organization

38

contrib

utors

htt

p:/

/ww

w.t

mm

i.org

/

The maturity of testing in The Netherlands

© S

YS

QA

Alm

ere

15

TMMi levels

1. Initial

2. Managed

Test Policy and StrategyTest PlanningTest Monitoring and ControlTest Design and ExecutionTest Environment

3. Defined

Test OrganizationTest Training ProgramTest Life Cycle and IntegrationNon-Functional TestingPeer Reviews

4. Management and Measurement

Test MeasurementSoftware Quality EvaluationAdvanced Peer Reviews

5. OptimizationDefect PreventionTest Process OptimizationQuality Control

© S

YS

QA

Alm

ere

16

How I got my data

• Goals• 20 organizations

• Telecommunication: 2• Government: 7• Financial services: 9• Industry: 2

• Approach assessments: quick scans• Analysis

© S

YS

QA

Alm

ere

17

Test policy and strategy - goals

• Establish a test policy• Establish a test strategy• Establish test performance indicators

© S

YS

QA

Alm

ere

18

Test policy and strategy: A: 30% V: 14%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200

10

20

30

40

50

60

70

80

90

100

Test policy and strategy

Organizations

Perc

en

tag

e o

f sp

ecifi

c g

oals

met

© S

YS

QA

Alm

ere

19

Test planning - goals• Perform a product risk assessment• Establish a test approach• Establish test estimates• Develop a test plan• Obtain commitment to the test plan

© S

YS

QA

Alm

ere

20

Test planning: A: 41%, V: 13%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200

10

20

30

40

50

60

70

80

90

100

Test planning

Organizations

Perc

en

tag

e o

f sp

ecifi

c g

oals

met

© S

YS

QA

Alm

ere

21

Test monitoring and control - goals

• Monitor test progress against plan• Monitor product quality against plan and

expectations• Manage corrective action to closure

© S

YS

QA

Alm

ere

22

Test monitoring and control: A: 47%, V: 15%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200

10

20

30

40

50

60

70

80

90

100

Test monitoring and control

Organizations

Perc

en

tag

e o

f sp

ecifi

c g

oals

met

© S

YS

QA

Alm

ere

23

Test design and execution - goals

• Perform test analysis and design using test design techniques

• Perform test implementation• Perform test execution• Manage test incidents to closure

© S

YS

QA

Alm

ere

24

Test design and execution: A: 60%, V: 8%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200

10

20

30

40

50

60

70

80

90

100

Test design and execution

Organizations

Perc

en

tag

e o

f sp

ecifi

c g

oals

met

© S

YS

QA

Alm

ere

25

Test environment - goals

• Develop test environment requirements• Perform test environment implementation• Manage and control test environments

© S

YS

QA

Alm

ere

26

Test environment: A: 59%, V: 9%

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 190

10

20

30

40

50

60

70

80

90

100

Test environment

Organizations

Perc

en

tag

e o

f sp

ecifi

c g

oals

met

© S

YS

QA

Alm

ere

27

Overview

Test

pol

icy

and

stra

tegy

Test

pla

nnin

g

Test

mon

itorin

g an

d co

ntro

l

Test

des

ign

and

exec

utio

n

Test

env

ironm

ent

0102030405060708090

100

Per process area

© S

YS

QA

Alm

ere

28

My conclusion

60%: Test design and execution

30%: Test policy and strategy

41%: Test planning

47%: Test monitoring and control

59%: Test environment

Inside

Outside

Our change approach is

insideout

What about Romanian testers;should they care about TMMi?

© S

YS

QA

Alm

ere

30

Reasons to care about TMMi

© S

YS

QA

Alm

ere

31

Outsourcing and the Allen curve

‘The distance between engineers’ offices affected the frequency of their communication’ – Prof. Thomas Allen MIT

© S

YS

QA

Alm

ere

32

8 meters

© S

YS

QA

Alm

ere

33

16 meters

© S

YS

QA

Alm

ere

34

80 kilometers

© S

YS

QA

Alm

ere

35

1600 kilometers

© S

YS

QA

Alm

ere

36

© S

YS

QA

Alm

ere

37

Email for questions, the documents I showed you or additional information:

jcannegieter@sysqa.nl