test maturity model integrated (tmmi) workshop software testing in systems- and software-engineering...

33
Listen | Challenge | Understand | Interpret | Create Private & Confidential Experimentus Ltd 85 Tottenham Court Road London W1T 4TQ T: +44 (0)870 770 6099 www.experimentus.com Implementing Test Maturity Model integrated (TMMi) Workshop Presented by: Geoff Thompson 22 nd September 2009 SIGiST

Upload: lethuy

Post on 20-Mar-2018

227 views

Category:

Documents


3 download

TRANSCRIPT

Listen | Challenge | Understand | Interpret | Create

Private & Confidential Experimentus Ltd 85 Tottenham Court Road London W1T 4TQ T: +44 (0)870 770 6099 www.experimentus.com

Implementing Test Maturity Model integrated

(TMMi) Workshop

Presented by:

Geoff Thompson

22nd September 2009SIGiST

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

Agenda

What is TMMi

Understanding of the TMMi level 2 process areas

TMMi Quick Assessment

A view of potential TMMi level

Copyright © 2009 Experimentus Ltd3

What is TMMi?

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

TMMi Model Purpose

4

A standard reference model to assess test process

Maturity

To identify:

– Testing strengths and best practices

– Project / test process issues and risks

– A realistic programme of test process improvements

– Facilities governance, measurement and accreditation of

processes

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

TMMi Model Scope

5

Supports software testing in systems- and software-

engineering

Addresses all levels of testing

– Low- and high-level dynamic testing are in scope

Provides a generic framework as a reference model for

test process improvement

Does not provide a process for test process

improvement

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

The TMMi Maturity Level Definitions

6

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

TMMi Structure

7

Maturity LevelMaturity Level

Testing Process

Capability

Indicates

Maturity Level

Process AreaTesting Process

Capability

Indicates Contains

Maturity Level

Process AreaTesting Process

Capability

Goal

Indicates Contains

Achieves

Maturity Level

Process Area

Specific &

Generic Practice

Testing Process

Capability

Goal

Indicates Contains

ContainsAchieves

Maturity Level

Process Area

Specific &

Generic Practice

Testing Process

Capability

Goal

Implementation

Indicates Contains

ContainsAchieves

Describes

Copyright © 2009 Experimentus Ltd8

Quick Assessment

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

Background

9

An indicative look at your test processes

Only works if you answer honestly

Focussed on Level 2

An indication of the outcome of a TMMi assessment

The process areas we will be reviewing:

– Test policy and strategy

– Test planning

– Test monitoring and control

– Test design and execution

– Test environment

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

Test Policy and Strategy – Improvement Goals

10

Test policy, aligned with the business (quality) policy, defined and

agreed

An organisation-wide test strategy established and deployed,

Goal-oriented test process performance indicators established

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

Test Policy and Strategy – Activity Questions

11

A test policy, aligned with the business (quality) policy, is defined

and agreed by the stakeholders

An organisation wide or programme wide test strategy is

established and deployed, identifying and defining test levels

Test process performance indicators defined and documented

Test process performance indicators are deployed

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

TMMi Quick Assessment (1)

12

Evaluation Dimensions

– Approach: commitment and ability to implement activity

– Deployment: how the activity is institutionalised

– Results: the effectiveness of the activity performed

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

TMMi Quick Assessment (2) – Process

Scoring based on all three evaluation dimensions

Activities performed in a Process Area scored between 0 and 10

Activity score is the average of the 3 dimensions

Process Area score is the average of all scores of the key

activities for that Process Area

13

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

Sample score sheet

14

Activities Score Test Policy and Strategy Level 2

Organisation: Date:

Activities performedDimension 0 1 2 3 4 5 6 7 8 9

1

0

Score

1. A test policy is defined and documented Approach

Deployment

Results

2. A test strategy is defined and documented Approach

Deployment

Results

3. Test process performance indicators are

defined and documentedApproach

Deployment

Results

4. Test process performance indicators are

deployedApproach

Deployment

Results

Total Score

Result (= total score / 4)

X

X

X

X

X

X

X

X

X

X

X

X

10

12

23

6

20/05/2009Acme Software Co Ltd

3.3

4

7.7

2

17

4.25

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

TMMi Quick Assessments – Result

For every Process Area:

– Average score :

Up to 2.99 “Not Satisfied”

Between 3 and 4.99 “Partly Satisfied”

Between 5 and 7.99 “Largely Satisfied”

Over 8 “Fully Satisfied”

15

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

Test Planning – Improvement Goals

Risk assessment performed to identify critical areas for testing

Test approach established and agreed

Well-founded test estimates established and maintained

Test plan established and maintained

Commitments to test plan established and maintained

16

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

Test Planning: Activity Questions

17

Risk assessment performed to identify critical areas for testing

Test approach based on risks is developed and agreed

Test estimates established / agreed according to documented

procedure

Test plan developed / agreed, ensuring that testing is well

managed and progress reported

Test project commitments made to external groups are

established / maintained

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

Test Monitoring and Control – Improvement Goals

18

Actual progress and performance is monitored against test plan

Actual product quality monitored against plan and expectations

Corrective actions managed to closure when progress or quality

deviate significantly from plan or expectations

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

Test Monitoring and Control – Activity Questions

19

Progress monitored against test plan

Product quality measured and reported in a standardised way

Changes to plan or expectations are controlled / managed

effectively

Data from previous projects used to shape future projects

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

Test Design and Execution – Improvement Goals

20

During test analysis and design test approach is translated into

tangible test conditions and test cases

Undertake all tasks required to prepare for test execution

Manage all incidents using the defined work flow

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

Test Design and Execution – Activity Questions

21

Test cases specified according to documented set of techniques

Test execution preparation tasks documented and completed

Test execution performed using documented procedures

Test incidents reported / managed using documented procedures

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

Test Environment – Improvement Goals

22

Environment requirements analysed, specified and verified as

achievable and appropriate

Environments and data are built, tested and provided according

to requirements

Environments and data appropriately supported

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

Test Environment – Activity Questions

23

Environment specified early in the project according to

documented procedure

Higher test levels performed in environment that is „real-life‟

Management and control of environment is by documented

procedures

Build, availability and usage of environment is co-ordinated using

documented procedure

Environment incidents reported according to a documented

procedure

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

The Result

Do you agree with the result?

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

How is everyone else doing?

25

Experimentus obtains accreditation for its Assessment Method

The need to understand the industry

Survey data collected during Q4 2008

More than 100 respondents

Many industry sectors

Respondents from 4 continents

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

Survey - Overall Results

26

TMMi level 172.5%

TMMi level 227.5%

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

A Reminder of Level 1

27

Level 1 means

– Little process

– Delivery of good testing

depends on people who

know what they are

doing

– The classic hero culture

of the indispensible

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

By Industry Sector

28

TMMi level 2

TMMi level 1

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

What are you going to do next?

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

Assessment Results

An assessment against TMMi Levels 2 and 3

30

0% 20% 40% 60% 80% 100%

Actively Performed

Mostly Deployed

Partially/Not Deployed

0% 20% 40% 60% 80% 100%

2.1 Testing & Debugging Goals & Policies

2.2 Test Planning & Execution

2.3 Test Design Techniques & Methods

3.1 Testing Organisation

3.2 Test Technical Training Program

3.3 Testing Integrated into the SDLC

3.4 Control & Monitoring of Testing Process

Key Process Areas Maturity Achievement

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd31

Thank you

www.experimentus.com

+44(0)8707 706099

[email protected]

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

About Experimentus....

32

An IT services consultancy providing organisations with software quality

management expertise

Key solutions aimed at optimising the approach to software quality

management

• Process assessment and certification

• Trusted advice and delivery

• Process improvement

• Metrics

Accredited TMMi method, assessors and lead assessors

Proven track record in using TMMi to help organisations improve

Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd

Acknowledgements

Test Maturity Model integrated (TMMi) – TMMi Foundation

CMM / CMMI – Software Engineering Institute (SEI)

TMMi Assessment Method, Experimentus Ltd