experiences with tmmi facts and figures - romanian testing conference 2014
TRANSCRIPT
Experiences with TMMi: facts and figures
Jan Jaap CannegieterVice President SYSQA B.V.
© S
YS
QA
Alm
ere
2
Agenda
0
20
40
60
80
100
What is
A maturity modeland
an assessment model
© S
YS
QA
Alm
ere
4
Test Maturity Model integration
Goals and practices
Certified assessors
Method independent
Organization
independent
Open source
© S
YS
QA
Alm
ere
5
TMMi levels
1. Initial
2. Managed
Test Policy and StrategyTest PlanningTest Monitoring and ControlTest Design and ExecutionTest Environment
3. Defined
Test OrganizationTest Training ProgramTest Life Cycle and IntegrationNon-Functional TestingPeer Reviews
4. Management and Measurement
Test MeasurementSoftware Quality EvaluationAdvanced Peer Reviews
5. OptimizationDefect PreventionTest Process OptimizationQuality Control
© S
YS
QA
Alm
ere
6
Formal a
ssessment Informal assessment
Planning
Data management
Process attribute rating
Reporting and output
© S
YS
QA
Alm
ere
9
Assessment type: Quick Scan
Test manager 1
Tester 1
QA manager
Test environment administrator
Test manager 2
Tester 2
Project manager
You
8.30 Opening & TMMi
9.00 Rating PA’s level 2
12.00 Lunch
13.00 Rating PA’s
14.00 Conclusions
14.30 Determine improvement actions
16.00 Making action plan
16.30 Evaluation
(20.30 On the couch watching TV)
© S
YS
QA
Alm
ere
10
The tool
© S
YS
QA
Alm
ere
11
TimeStart
I I IQS QS QS QS QS
F
QS
Assessments and TMMi implementations
Level 2
© S
YS
QA
Alm
ere
12
Assessment types compared
0 100
Formal Informal Quick scan
Accuracy
Costs
100% Use at SYSQA
© S
YS
QA
Alm
ere
13
TMMi today
TMMi model
TAMAR
F i v e s e r v i c e p r o v i d e r s
i n f o u r c o u n t r i e s
22 assessors
1000 members
16 supporting organization
38
contrib
utors
htt
p:/
/ww
w.t
mm
i.org
/
The maturity of testing in The Netherlands
© S
YS
QA
Alm
ere
15
TMMi levels
1. Initial
2. Managed
Test Policy and StrategyTest PlanningTest Monitoring and ControlTest Design and ExecutionTest Environment
3. Defined
Test OrganizationTest Training ProgramTest Life Cycle and IntegrationNon-Functional TestingPeer Reviews
4. Management and Measurement
Test MeasurementSoftware Quality EvaluationAdvanced Peer Reviews
5. OptimizationDefect PreventionTest Process OptimizationQuality Control
© S
YS
QA
Alm
ere
16
How I got my data
• Goals• 20 organizations
• Telecommunication: 2• Government: 7• Financial services: 9• Industry: 2
• Approach assessments: quick scans• Analysis
© S
YS
QA
Alm
ere
17
Test policy and strategy - goals
• Establish a test policy• Establish a test strategy• Establish test performance indicators
© S
YS
QA
Alm
ere
18
Test policy and strategy: A: 30% V: 14%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200
10
20
30
40
50
60
70
80
90
100
Test policy and strategy
Organizations
Perc
en
tag
e o
f sp
ecifi
c g
oals
met
© S
YS
QA
Alm
ere
19
Test planning - goals• Perform a product risk assessment• Establish a test approach• Establish test estimates• Develop a test plan• Obtain commitment to the test plan
© S
YS
QA
Alm
ere
20
Test planning: A: 41%, V: 13%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200
10
20
30
40
50
60
70
80
90
100
Test planning
Organizations
Perc
en
tag
e o
f sp
ecifi
c g
oals
met
© S
YS
QA
Alm
ere
21
Test monitoring and control - goals
• Monitor test progress against plan• Monitor product quality against plan and
expectations• Manage corrective action to closure
© S
YS
QA
Alm
ere
22
Test monitoring and control: A: 47%, V: 15%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200
10
20
30
40
50
60
70
80
90
100
Test monitoring and control
Organizations
Perc
en
tag
e o
f sp
ecifi
c g
oals
met
© S
YS
QA
Alm
ere
23
Test design and execution - goals
• Perform test analysis and design using test design techniques
• Perform test implementation• Perform test execution• Manage test incidents to closure
© S
YS
QA
Alm
ere
24
Test design and execution: A: 60%, V: 8%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 200
10
20
30
40
50
60
70
80
90
100
Test design and execution
Organizations
Perc
en
tag
e o
f sp
ecifi
c g
oals
met
© S
YS
QA
Alm
ere
25
Test environment - goals
• Develop test environment requirements• Perform test environment implementation• Manage and control test environments
© S
YS
QA
Alm
ere
26
Test environment: A: 59%, V: 9%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 190
10
20
30
40
50
60
70
80
90
100
Test environment
Organizations
Perc
en
tag
e o
f sp
ecifi
c g
oals
met
© S
YS
QA
Alm
ere
27
Overview
Test
pol
icy
and
stra
tegy
Test
pla
nnin
g
Test
mon
itorin
g an
d co
ntro
l
Test
des
ign
and
exec
utio
n
Test
env
ironm
ent
0102030405060708090
100
Per process area
© S
YS
QA
Alm
ere
28
My conclusion
60%: Test design and execution
30%: Test policy and strategy
41%: Test planning
47%: Test monitoring and control
59%: Test environment
Inside
Outside
Our change approach is
insideout
What about Romanian testers;should they care about TMMi?
© S
YS
QA
Alm
ere
30
Reasons to care about TMMi
© S
YS
QA
Alm
ere
31
Outsourcing and the Allen curve
‘The distance between engineers’ offices affected the frequency of their communication’ – Prof. Thomas Allen MIT
© S
YS
QA
Alm
ere
32
8 meters
© S
YS
QA
Alm
ere
33
16 meters
© S
YS
QA
Alm
ere
34
80 kilometers
© S
YS
QA
Alm
ere
35
1600 kilometers
© S
YS
QA
Alm
ere
36
© S
YS
QA
Alm
ere
37
Email for questions, the documents I showed you or additional information: