tester: get out of your cave!' by jan jaap cannegieter
DESCRIPTION
The levels 2 and 3 of TMMi, the successor of TMM, are launched in 2008. By now, the company of the presenter performed almost 20 assessments, most informal assessments. Assessments where performed in telecom, utilities, financial services and (semi) government. The first conclusions about the maturity of testing (in the Netherlands) can be drawn. In the presentation the structure of TMMi will be introduced briefly, as well as the way the assessments where done. Then the results will be presented, as well as the conclusion. The overall picture is that test process improvement is something testers do in there cave. To improve testing rigorous we have to get out of our cave. At the end of the presentation instructions are given how to do this.TRANSCRIPT
Tester, get out of your cave!
Jan Jaap Cannegieter
Vice President SYSQA B.V.
© S
YS
QA
Alm
ere
3
Why are we here?
© S
YS
QA
Alm
ere
4
Agenda
• TMMi in under 10 minutes
• Data from 20 TMMi workshops
• Conclusions
• How to get out of your cave
© S
YS
QA
Alm
ere
6
TMMi – essentials
• Test maturity framework• Benchmark model for assessments• 5 levels• Based on:
• TMM• Evolutionary Testmodel (Gelperin / Hetzel)• CMMI for Development
• In line with ISTQB terminology• Developed by the TMMi-foundation• Public domain• www.tmmifoundation.org
© S
YS
QA
Alm
ere
7
TMMi levels
1. Initial
2. Managed
Test Policy and StrategyTest PlanningTest Monitoring and ControlTest Design and ExecutionTest Environment
3. Defined
Test OrganizationTest Training ProgramTest Life Cycle and IntegrationNon-Functional TestingPeer Reviews
4. Management and Measurement
Test MeasurementSoftware Quality EvaluationAdvanced Peer Reviews
5. OptimizationDefect PreventionTest Process OptimizationQuality Control
© S
YS
QA
Alm
ere
8
How I got my data
• Goals• Learn
• Presales
• Assignments
• 20 organizations• Telecommunication: 2
• Government: 7
• Financial services: 9
• Industry: 2
• Analysis
© S
YS
QA
Alm
ere
9
Quick scans – a demo
Results
Fasten your seatbelts
© S
YS
QA
Alm
ere
11
Test policy and strategy - goals
• Establish a test policy
• Establish a test strategy
• Establish test performance indicators
© S
YS
QA
Alm
ere
12
Test policy and strategy: A: 30% V: 14%
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Test policy and strategy
Organizations
Percen
tag
e o
f sp
ecif
ic g
oals
met
© S
YS
QA
Alm
ere
13
Test planning - goals
• Perform a product risk assessment
• Establish a test approach
• Establish test estimates
• Develop a test plan
• Obtain commitment to the test plan
© S
YS
QA
Alm
ere
14
Test planning: A: 41%, V: 13%
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Test planning
Organizations
Percen
tag
e o
f sp
ecif
ic g
oals
met
© S
YS
QA
Alm
ere
15
Test monitoring and control - goals
• Monitor test progress against plan
• Monitor product quality against plan and expectations
• Manage corrective action to closure
© S
YS
QA
Alm
ere
16
Test monitoring and control: A: 47%, V: 15%
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Test monitoring and control
Organizations
Percen
tag
e o
f sp
ecif
ic g
oals
met
© S
YS
QA
Alm
ere
17
Test design and execution - goals
• Perform test analysis and design using test design techniques
• Perform test implementation
• Perform test execution
• Manage test incidents to closure
© S
YS
QA
Alm
ere
18
Test design and execution: A: 60%, V: 8%
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Test design and execution
Organizations
Percen
tag
e o
f sp
ecif
ic g
oals
met
© S
YS
QA
Alm
ere
19
Test environment - goals
• Develop test environment requirements
• Perform test environment implementation
• Manage and control test environments
© S
YS
QA
Alm
ere
20
Test environment: A: 59%, V: 9%
0
10
20
30
40
50
60
70
80
90
100
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Test environment
Organizations
Percen
tag
e o
f sp
ecif
ic g
oals
met
© S
YS
QA
Alm
ere
21
Overview
0
10
20
30
40
50
60
70
80
90
100
Per process area
© S
YS
QA
Alm
ere
22
My conclusion
60%: Test design and execution
30%: Test policy and strategy
41%: Test planning
47%: Test monitoring and control
59%: Test environment
Inside
Outside
© S
YS
QA
Alm
ere
23
We are in a cave!
© S
YS
QA
Alm
ere
24
© S
YS
QA
Alm
ere
25
In case you disagree…
© S
YS
QA
Alm
ere
26
Best practices ‘outside of your cave’
• Perform a product risk assessment and establish a test approach with relevant stakeholders
• Obtain commitment for your plan from relevant stakeholders
• Plan stakeholder involvement in your project
• Monitor stakeholder involvement in your project
• Make release advice with people outside of testing
• Establish and maintain a test strategy with relevant management
• Define, measure and report test performance indicators to relevant stakeholders
• Establish and maintain a test policy with higher management (business and IT)
Stakeholder management
27
The basics
© S
YS
QA
Alm
ere
28
Stakeholder
A person or group of persons who will be involved in or effected by your test project or improvement project
© S
YS
QA
Alm
ere
29
Stakeholder management
• Stakeholder identification
• Stakeholder analysis
• Stakeholder involvement
© S
YS
QA
Alm
ere
30
Stakeholder aspects
• Organizational level
• Power
• Business interest
• Mindset
• Ambition
• Behavioral style
• Stakeholder type
© S
YS
QA
Alm
ere
31
Type of stakeholders
Demanding Discretionary
Sleeping
Dependent
Dangerous Dominant
Definitive
Urgency Legitimacy
Power
Mitchell, Agle, Wood, 1997
© S
YS
QA
Alm
ere
32
Stakeholdercard – an example
Please think this over
I’ll turn on some inspiring music
© S
YS
QA
Alm
ere
34
© S
YS
QA
Alm
ere
35
© S
YS
QA
Alm
ere
36
Stuart Reid’s test competence model for testers
Test skills
IT skills
Domain knowledge
Soft skills
© S
YS
QA
Alm
ere
37
My test competence model for test consultants
Test skills
IT skills
Domain knowledge
Soft skills
Political skills
© S
YS
QA
Alm
ere
38
© S
YS
QA
Alm
ere
41