test maturity model integrated (tmmi) workshop software testing in systems- and software-engineering...
TRANSCRIPT
Listen | Challenge | Understand | Interpret | Create
Private & Confidential Experimentus Ltd 85 Tottenham Court Road London W1T 4TQ T: +44 (0)870 770 6099 www.experimentus.com
Implementing Test Maturity Model integrated
(TMMi) Workshop
Presented by:
Geoff Thompson
22nd September 2009SIGiST
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
Agenda
What is TMMi
Understanding of the TMMi level 2 process areas
TMMi Quick Assessment
A view of potential TMMi level
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
TMMi Model Purpose
4
A standard reference model to assess test process
Maturity
To identify:
– Testing strengths and best practices
– Project / test process issues and risks
– A realistic programme of test process improvements
– Facilities governance, measurement and accreditation of
processes
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
TMMi Model Scope
5
Supports software testing in systems- and software-
engineering
Addresses all levels of testing
– Low- and high-level dynamic testing are in scope
Provides a generic framework as a reference model for
test process improvement
Does not provide a process for test process
improvement
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
The TMMi Maturity Level Definitions
6
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
TMMi Structure
7
Maturity LevelMaturity Level
Testing Process
Capability
Indicates
Maturity Level
Process AreaTesting Process
Capability
Indicates Contains
Maturity Level
Process AreaTesting Process
Capability
Goal
Indicates Contains
Achieves
Maturity Level
Process Area
Specific &
Generic Practice
Testing Process
Capability
Goal
Indicates Contains
ContainsAchieves
Maturity Level
Process Area
Specific &
Generic Practice
Testing Process
Capability
Goal
Implementation
Indicates Contains
ContainsAchieves
Describes
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
Background
9
An indicative look at your test processes
Only works if you answer honestly
Focussed on Level 2
An indication of the outcome of a TMMi assessment
The process areas we will be reviewing:
– Test policy and strategy
– Test planning
– Test monitoring and control
– Test design and execution
– Test environment
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
Test Policy and Strategy – Improvement Goals
10
Test policy, aligned with the business (quality) policy, defined and
agreed
An organisation-wide test strategy established and deployed,
Goal-oriented test process performance indicators established
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
Test Policy and Strategy – Activity Questions
11
A test policy, aligned with the business (quality) policy, is defined
and agreed by the stakeholders
An organisation wide or programme wide test strategy is
established and deployed, identifying and defining test levels
Test process performance indicators defined and documented
Test process performance indicators are deployed
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
TMMi Quick Assessment (1)
12
Evaluation Dimensions
– Approach: commitment and ability to implement activity
– Deployment: how the activity is institutionalised
– Results: the effectiveness of the activity performed
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
TMMi Quick Assessment (2) – Process
Scoring based on all three evaluation dimensions
Activities performed in a Process Area scored between 0 and 10
Activity score is the average of the 3 dimensions
Process Area score is the average of all scores of the key
activities for that Process Area
13
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
Sample score sheet
14
Activities Score Test Policy and Strategy Level 2
Organisation: Date:
Activities performedDimension 0 1 2 3 4 5 6 7 8 9
1
0
Score
1. A test policy is defined and documented Approach
Deployment
Results
2. A test strategy is defined and documented Approach
Deployment
Results
3. Test process performance indicators are
defined and documentedApproach
Deployment
Results
4. Test process performance indicators are
deployedApproach
Deployment
Results
Total Score
Result (= total score / 4)
X
X
X
X
X
X
X
X
X
X
X
X
10
12
23
6
20/05/2009Acme Software Co Ltd
3.3
4
7.7
2
17
4.25
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
TMMi Quick Assessments – Result
For every Process Area:
– Average score :
Up to 2.99 “Not Satisfied”
Between 3 and 4.99 “Partly Satisfied”
Between 5 and 7.99 “Largely Satisfied”
Over 8 “Fully Satisfied”
15
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
Test Planning – Improvement Goals
Risk assessment performed to identify critical areas for testing
Test approach established and agreed
Well-founded test estimates established and maintained
Test plan established and maintained
Commitments to test plan established and maintained
16
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
Test Planning: Activity Questions
17
Risk assessment performed to identify critical areas for testing
Test approach based on risks is developed and agreed
Test estimates established / agreed according to documented
procedure
Test plan developed / agreed, ensuring that testing is well
managed and progress reported
Test project commitments made to external groups are
established / maintained
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
Test Monitoring and Control – Improvement Goals
18
Actual progress and performance is monitored against test plan
Actual product quality monitored against plan and expectations
Corrective actions managed to closure when progress or quality
deviate significantly from plan or expectations
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
Test Monitoring and Control – Activity Questions
19
Progress monitored against test plan
Product quality measured and reported in a standardised way
Changes to plan or expectations are controlled / managed
effectively
Data from previous projects used to shape future projects
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
Test Design and Execution – Improvement Goals
20
During test analysis and design test approach is translated into
tangible test conditions and test cases
Undertake all tasks required to prepare for test execution
Manage all incidents using the defined work flow
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
Test Design and Execution – Activity Questions
21
Test cases specified according to documented set of techniques
Test execution preparation tasks documented and completed
Test execution performed using documented procedures
Test incidents reported / managed using documented procedures
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
Test Environment – Improvement Goals
22
Environment requirements analysed, specified and verified as
achievable and appropriate
Environments and data are built, tested and provided according
to requirements
Environments and data appropriately supported
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
Test Environment – Activity Questions
23
Environment specified early in the project according to
documented procedure
Higher test levels performed in environment that is „real-life‟
Management and control of environment is by documented
procedures
Build, availability and usage of environment is co-ordinated using
documented procedure
Environment incidents reported according to a documented
procedure
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
The Result
Do you agree with the result?
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
How is everyone else doing?
25
Experimentus obtains accreditation for its Assessment Method
The need to understand the industry
Survey data collected during Q4 2008
More than 100 respondents
Many industry sectors
Respondents from 4 continents
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
Survey - Overall Results
26
TMMi level 172.5%
TMMi level 227.5%
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
A Reminder of Level 1
27
Level 1 means
– Little process
– Delivery of good testing
depends on people who
know what they are
doing
– The classic hero culture
of the indispensible
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
By Industry Sector
28
TMMi level 2
TMMi level 1
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
Assessment Results
An assessment against TMMi Levels 2 and 3
30
0% 20% 40% 60% 80% 100%
Actively Performed
Mostly Deployed
Partially/Not Deployed
0% 20% 40% 60% 80% 100%
2.1 Testing & Debugging Goals & Policies
2.2 Test Planning & Execution
2.3 Test Design Techniques & Methods
3.1 Testing Organisation
3.2 Test Technical Training Program
3.3 Testing Integrated into the SDLC
3.4 Control & Monitoring of Testing Process
Key Process Areas Maturity Achievement
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd31
Thank you
www.experimentus.com
+44(0)8707 706099
Copyright © 2009 Experimentus LtdCopyright © 2009 Experimentus Ltd
About Experimentus....
32
An IT services consultancy providing organisations with software quality
management expertise
Key solutions aimed at optimising the approach to software quality
management
• Process assessment and certification
• Trusted advice and delivery
• Process improvement
• Metrics
Accredited TMMi method, assessors and lead assessors
Proven track record in using TMMi to help organisations improve