tester: get out of your cave!' by jan jaap cannegieter

Tester, get out of your cave!

Jan Jaap Cannegieter

Vice President SYSQA B.V.

© S

YS

QA

Alm

ere

2

© S

YS

QA

Alm

ere

3

Why are we here?

© S

YS

QA

Alm

ere

4

Agenda

• TMMi in under 10 minutes

• Data from 20 TMMi workshops

• Conclusions

• How to get out of your cave

© S

YS

QA

Alm

ere

6

TMMi – essentials

• Test maturity framework• Benchmark model for assessments• 5 levels• Based on:

• TMM• Evolutionary Testmodel (Gelperin / Hetzel)• CMMI for Development

• In line with ISTQB terminology• Developed by the TMMi-foundation• Public domain• www.tmmifoundation.org

© S

YS

QA

Alm

ere

7

TMMi levels

1. Initial

2. Managed

Test Policy and StrategyTest PlanningTest Monitoring and ControlTest Design and ExecutionTest Environment

3. Defined

Test OrganizationTest Training ProgramTest Life Cycle and IntegrationNon-Functional TestingPeer Reviews

4. Management and Measurement

Test MeasurementSoftware Quality EvaluationAdvanced Peer Reviews

5. OptimizationDefect PreventionTest Process OptimizationQuality Control

© S

YS

QA

Alm

ere

8

How I got my data

• Goals• Learn

• Presales

• Assignments

• 20 organizations• Telecommunication: 2

• Government: 7

• Financial services: 9

• Industry: 2

• Analysis

© S

YS

QA

Alm

ere

9

Quick scans – a demo

Results

Fasten your seatbelts

© S

YS

QA

Alm

ere

11

Test policy and strategy - goals

• Establish a test policy

• Establish a test strategy

• Establish test performance indicators

© S

YS

QA

Alm

ere

12

Test policy and strategy: A: 30% V: 14%

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Test policy and strategy

Organizations

Percen

tag

e o

f sp

ecif

ic g

oals

met

© S

YS

QA

Alm

ere

13

Test planning - goals

• Perform a product risk assessment

• Establish a test approach

• Establish test estimates

• Develop a test plan

• Obtain commitment to the test plan

© S

YS

QA

Alm

ere

14

Test planning: A: 41%, V: 13%

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Test planning

Organizations

Percen

tag

e o

f sp

ecif

ic g

oals

met

© S

YS

QA

Alm

ere

15

Test monitoring and control - goals

• Monitor test progress against plan

• Monitor product quality against plan and expectations

• Manage corrective action to closure

© S

YS

QA

Alm

ere

16

Test monitoring and control: A: 47%, V: 15%

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Test monitoring and control

Organizations

Percen

tag

e o

f sp

ecif

ic g

oals

met

© S

YS

QA

Alm

ere

17

Test design and execution - goals

• Perform test analysis and design using test design techniques

• Perform test implementation

• Perform test execution

• Manage test incidents to closure

© S

YS

QA

Alm

ere

18

Test design and execution: A: 60%, V: 8%

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

Test design and execution

Organizations

Percen

tag

e o

f sp

ecif

ic g

oals

met

© S

YS

QA

Alm

ere

19

Test environment - goals

• Develop test environment requirements

• Perform test environment implementation

• Manage and control test environments

© S

YS

QA

Alm

ere

20

Test environment: A: 59%, V: 9%

0

10

20

30

40

50

60

70

80

90

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Test environment

Organizations

Percen

tag

e o

f sp

ecif

ic g

oals

met

© S

YS

QA

Alm

ere

21

Overview

0

10

20

30

40

50

60

70

80

90

100

Per process area

© S

YS

QA

Alm

ere

22

My conclusion

60%: Test design and execution

30%: Test policy and strategy

41%: Test planning

47%: Test monitoring and control

59%: Test environment

Inside

Outside

© S

YS

QA

Alm

ere

23

We are in a cave!

© S

YS

QA

Alm

ere

24

© S

YS

QA

Alm

ere

25

In case you disagree…

© S

YS

QA

Alm

ere

26

Best practices ‘outside of your cave’

• Perform a product risk assessment and establish a test approach with relevant stakeholders

• Obtain commitment for your plan from relevant stakeholders

• Plan stakeholder involvement in your project

• Monitor stakeholder involvement in your project

• Make release advice with people outside of testing

• Establish and maintain a test strategy with relevant management

• Define, measure and report test performance indicators to relevant stakeholders

• Establish and maintain a test policy with higher management (business and IT)

Stakeholder management

27

The basics

© S

YS

QA

Alm

ere

28

Stakeholder

A person or group of persons who will be involved in or effected by your test project or improvement project

© S

YS

QA

Alm

ere

29

Stakeholder management

• Stakeholder identification

• Stakeholder analysis

• Stakeholder involvement

© S

YS

QA

Alm

ere

30

Stakeholder aspects

• Organizational level

• Power

• Business interest

• Mindset

• Ambition

• Behavioral style

• Stakeholder type

© S

YS

QA

Alm

ere

31

Type of stakeholders

Demanding Discretionary

Sleeping

Dependent

Dangerous Dominant

Definitive

Urgency Legitimacy

Power

Mitchell, Agle, Wood, 1997

© S

YS

QA

Alm

ere

32

Stakeholdercard – an example

Please think this over

I’ll turn on some inspiring music

© S

YS

QA

Alm

ere

34

© S

YS

QA

Alm

ere

35

© S

YS

QA

Alm

ere

36

Stuart Reid’s test competence model for testers

Test skills

IT skills

Domain knowledge

Soft skills

© S

YS

QA

Alm

ere

37

My test competence model for test consultants

Test skills

IT skills

Domain knowledge

Soft skills

Political skills

© S

YS

QA

Alm

ere

38

© S

YS

QA

Alm

ere

40

Email for the stakeholder card:

jcannegieter@sysqa.nl

© S

YS

QA

Alm

ere

41