tmmi implementation guideline

Path to reach Level 3 test

process maturity in TMMi

framework

M C Nagaraj

[email protected]

in.linkedin.com/in/mcnagaraj

mailto:[email protected]

Copyright STAG Software Pvt. Ltd., 2011

Agenda

TMMi – Level 1 to 5

Requirement interpretation

Linking current process to TMMi Specific practices

All TMMi related information are taken from TMMi framework document V 3.1 created and Copyrights owned by TMMi foundation.www.tmmifoundation.org


Test Measurement

Software Quality Evaluation

Advanced Peer Reviews

Defect Prevention

Total Process Optimization

Quality Control

Test Policy and Strategy, Test Planning, Test Monitoring and

Control, Test Design and Execution, Test Environment

Test Organization, Test Training Program, Test Lifecycle and

Integration, Non-functional Testing, Peer Reviews

Level 1 : Initial

TMMi Framework

Five levels of Testing Process Maturity

Level 2 : Managed

Level 3 : Defined

Level 4 : Measured

Level 5 : Optimization


Level-1

• Testing is a chaotic process

• Not distinguished from debugging

• Documented set of specification not available

• Tests are developed in ad-hoc way after coding is completed

• The objective of testing is to show that software works


Test Measurement



Defect Prevention


Quality Control





Level 1 : Initial

TMMi Level 2


Level 2 : Managed

Level 3 : Defined

Level 4 : Measured


Process areas at TMMi Level 2

2.1 Test Policy and strategy

2.2 Test Planning

2.3 Test Monitoring and Control

2.4 Test Design and Execution

2.5 Test Environment


2.1 Test Policy and Strategy

Specific Goals Specific Practices Guidelines to implement

SG 1 - Establish a Test Policy SP 1.1 Define test goalsSP 1.2 Define test policySP 1.3 Distribute the test policy to stakeholders

-Look at organization business goaland role of testing in achieving the same- Policy should address test cycle activities are planned, done by trained engineers, process improve with management support/focus

SG 2 - Establish a Test Strategy SP 2.1 Perform generic product risk assessmentSP 2.2 Define test strategySP 2.3 Distribute the test strategy to stakeholders

-Use organization standard way ofidentifying risks in a project-Define quality levels and objectives

SG 3 - Establish Test Performance Indicators

SP 3.1 Define test performance indicatorsSP 3.2 Deploy test performanceindicators

-Define how you measure success of test process-Set some target for key metrics and measure what you achieved




2.2 Test Planning





2.2 Test Planning


SG 1- Perform Product Risk Assessment

SP 1.1 Define product risk categoriesSP 1.2 Identify product risksSP 1.3 Analyze product risks

-Look at from end users perspective value & prioritize requirements-Can you predict what can upset end users?

SG 2 - Establish a Test Approach SP 2.1 Identify items and features to be testedSP 2.2 Define the test approachSP 2.3 Define entry criteriaSP 2.4 Define exit criteriaSP 2.5 Define suspension and resumption criteria

- Extract list of features-Extract list of user types and business use cases they use-Extract potential defect types-Quality levels and types of defects to be uncovered at each levels-Entry, exit, suspension and resumption criteria for all levels

SG 3 - Establish Test Estimates SP 3.1 Establish a top-level work breakdown structureSP 3.2 Define test life cycleSP 3.3 Determine estimates for test effort and cost

-Have organization standard test life cycle activity list-Customize this to project specific-Estimate effort, review update and baseline-Track number of cycles of re-estimation done


2.2 Test Planning


SG 4 – Develop a Test Plan SP 4.1 Establish the test scheduleSP 4.2 Plan for test staffingSP 4.3 Plan for stakeholders involvementSP 4.4 Identify test project risksSP 4.5 Establish the test plan

-Milestone definition-Detail schedule-Risk assessment and prioritization-Plan document with important sections in place

SG 5 – Obtain Commitment to the Test Plan

SP 5.1 Review test planSP 5.2 Reconcile work and resource levelsSP 5.3 Obtain test plan commitments

-Review plan with all concerned-Update feedback and ensure confidence is high to meet the plan-Team commitments for all dependency will help




2.2 Test Planning







SG 1 – Monitor Test Progress against Plan SP 1.1 Monitor test planning parametersSP 1.2 Monitor test environment resources provided and usedSP 1.3 Monitor test commitmentsSP 1.4 Monitor test project risksSP 1.5 Monitor stakeholder involvementSP 1.6 Conduct test progress reviewsSP 1.7 Conduct test progress milestone reviews

-Monitor effort/cost, schedule variance against plan-Ensure test environment is same as stated in expectation in document-Milestone review, weekly review etc.-Stakeholders update on key milestone and seek feedback-Risk tracking

SG 2 – Monitor Product Quality against Plan and Expectations

SP 2.1 Check against entry criteriaSP 2.2 Monitor defectsSP 2.3 Monitor product risksSP 2.4 Monitor exit criteriaSP 2.5 Monitor suspension and resumption criteriaSP 2.6 Conduct product quality reviewsSP 2.7 Conduct product quality milestone reviews

-Quality levels exit criteria defined and actual- Monitor suspension and resumptions cycles- Test summary report at key quality levels achievement



Specific Goals Specific Practices Guideline to implement

SG 3 – Manage Corrective Action to Closure

SP 3.1 Analyze issuesSP 3.2 Take corrective actionSP 3.3 Manage corrective action

-Track project issues formally-Track action items and ensure closure




2.2 Test Planning







SG 1 – Perform Test Analysis and Design using Test Design Techniques

SP 1.1 Identify and prioritize test conditionsSP 1.2 Identify and prioritize test casesSP 1.3 Identify necessary specific test dataSP 1.4 Maintain horizontal traceability with requirements

-Have a clear test case architecture to be followed across organization-Organize test cases at quality level -Maintain requirement traceability to test scenarios/test cases, fault type, types of test applicable

SG 2 – Perform Test Implementation

SP 2.1 Develop and prioritize test proceduresSP 2.2 Create specific test dataSP 2.3 Specify intake test procedureSP 2.4 Develop test execution schedule

-Test cycle schedule-Build health check criteria-Create re-usable test data- Quality level entry/exit criteria




SG 3 – Perform Test Execution SP 3.1 Perform intake testSP 3.2 Execute test casesSP 3.3 Report test incidentsSP 3.4 Write test log

-Verify entry criteria beforestarting any levels of testing- Execute test as per plan and record outcome-Capture defects to track and close

SG 4 – Manage test incident to closure

SP 4.1 Decide deposition of test incidents in configuration control boardSP 4.2 Perform appropriate action to close the test incidentSP 4.3 Track the status of test incidents

-Be part of change control board to resolve all issues reported during testing- Ensure defects fixed are verified and closed in each release- Defect tracking report in various forms so that helps to take right decision




2.2 Test Planning







SG 1 – Develop Test Environment Requirements

SP 1.1 Elicit test environments needsSP 1.2 Develop the test environment requirementsSP 1.3 Analyze the test environment requirements

-Capture required environment clearly at one place- Ensure environment available and required are identified

SG 2 – Perform Test Environment Implementation

SP 2.1 Implement the test environmentSP 2.2 Create generic test dataSP 2.3 Specify test environment intake test procedureSP 2.4 Perform test environment intake test

-Set up test environment as needed- Create generic test data and archive for re-use- Verify whether needs of environment required for testing is met formally




SG 3 – Manage and Control Test Environments

SP 3.1 Perform systemsmanagementSP 3.2 Perform test data managementSP 3.3 Co-ordinate the availability and usage of the test environmentsSP 3.4 Report and manage test environment incidents

-Plan and track system requirements-Plan development of test data and ensure it is version controlled- Ensure required co-ordination process in place when environment are shared- If environment issues found capture and track till it is closed



What you can see in L2 organization?

• Organization test goals and policy published

• There is a clear separation between debugging and testing phase

• It is a planned activity in project plan

• Plan starts after coding is complete (only execution based testing visible)

• Basic testing techniques in place

• Testing is multi-leveled

• Test environment managed as required


Test Measurement



Defect Prevention


Quality Control





Level 1 : Initial

TMMi Framework


Level 2 : Managed

Level 3 : Defined

Level 4 : Measured



3.1 Test Organization

3.2 Test Training Program

3.3 Test Lifecycle and Integration

3.4 Non-functional testing

3.5 Peer Reviews




SG 1 – Establish a Test Organization

SP 1.1 Define the test organizationSP 1.2 Obtain commitment for the test organizationSP 1.3 Implement the test organization

-Clear definition of test team organization- Management commitment to support the test organization- Create team to fill all roles

SG 2 – Establish Test Functions for Test Specialists

SP 2.1 Identify test functionsSP 2.2 Develop job descriptionsSP 2.3 Assign staff members to test functions

-Define key test lifecycle activities-Map role and responsibilities to key test life cycle activities

SG 3 – Establish Test Career Paths

SP 3.1 Establish test career pathSP 3.2 Develop personal test career development plans

-Ensure there is clear definition of career path for test engineers- Ensure individual career plans prepared and tracked by managers




SG 4 – Determine, Plan and Implement Test Process Improvement

SP 4.1 Assess the organization’s test processSP 4.2 Identify the organization test process improvementsSP 4.3 Plan test process improvementsSP 4.4 Implement test process improvements

-Assess organization test process formally- Identify improvement areas- Plan test process improvements by having a team- Ensure team work as per plan and achieve improvements planned

SG 5 – Deploy the Organization Test Process and Incorporate Lessons Learned

SP 5.1 Deploy standard test process and test process assetsSP 5.2 Monitor implementationSP 5.3 Incorporate lessons learnt into the organizational test process

-Define standard process- Capture test process assets- Monitor adherence to process by planned audits- Analyze completed projects and recommend any improvement to process needed


3.2 Test Training Program


SG 1 – Establish an Organizational Test Training Capability

SP 1.1 Establish the strategictest training needsSP 1.2 Align the organizational and project test training needsSP 1.3 Establish an organizational test training planSP 1.4 Establish test training capability

-Define role based competency and standard training needs- Identify project specific competency required and plan training- Have clear training plan for team and project specific training on need basis-Ensure trainers internal/external identified and ready for all topics

SG 2 – Provide Necessary Test Training

SP 2.1 Deliver test trainingSP 2.2 Establish test training recordsSP 2.3 Assess test training effectiveness

-Conduct training as per plan- Maintain test training records- Verify the effectiveness of training conducted on job




SG 1 – Establish Organizational Test Process and Assets

SP 1.1 Establish standard test processSP 1.2 Establish test lifecycle model descriptions addressing all test levelsSP 1.3 Establish tailoring criteria and guidelinesSP 1.4 Establish the organization’s testprocess databaseSP 1.5 Establish the organization's test process asset librarySP 1.6 Establish work environment standards

-Have organization standard test process approved- Define standard test lifecycle activities-Establish tailoring guidelines and criteria-Capture test process followed at project level as data - Capture process assets including tailoring done- Ensure work environment is defined and adhered to as per plan

SG 2 – Integrate Test Lifecycle Models with Development Models

SP 2.1 Establish integrated test lifecycle modelsSP 2.2 Review integrated lifecycle modelsSP 2.3 Obtain commitments on the role of testing within the integrated lifecycle models

-Ensure multiple test cycles are planned with focus on all levels of testing- Review with all concerned on depends across each team and agree on entry/exit criteria




SG 3 – Establish a Master Test Plan

SP 3.1 Perform product risk assessmentSP 3.2 Establish the test approachSP 3.3 Establish test estimatesSP 3.4 Define the organization for testingSP 3.5 Develop the master test planSP 3.6 Obtain commitment to the master test plan

-Prioritize customer expectation- Define test strategy- Estimate effort required- Define team required -Have master test plan with milestone- Have detail plan for all milestone- Have commitment from all concerned in team on schedule


3.4 Non-functional Testing


SG 1 – Perform a Non-Functional Product Risk Assessment

SP 1.1 Identify non-functional product risksSP 1.2 Analyze non-functional product risks

-Identify non-functionalproduct requirements- Analyze what can go wrong here

SG 2 – Establish a Non-Functional Test Approach

SP 2.1 Identify features to be testedSP 2.2 Define the non-functional test approachSP 2.3 Define non-functional exit criteria

-Identify scope for NF testing- Define test approach for NF testing- Define exit criteria for all planned NF tests


3.4 Non-functional Testing


SG 3 – Perform Non-Functional Test Analysis and Design

SP 3.1 Identify and prioritize non-functional test conditionsSP 3.2 Identify and prioritize non-functional test casesSP 3.3 Identify necessary specific test dataSP 3.4 Maintain horizontal traceability with non-functionalrequirements

-Identify use cases where NF requirements may not be fully met- Have scenarios and tests cases to replicate the identified use case- Ensure NF requirement to test assets traceability is established

SG 4 – Perform Non-Functional Test Implementation

SP 4.1 Develop and prioritize non-functional test requirementsSP 4.2 Create specific test data

-Scope and plan for NF testing-Create required scripts to produce the targeted use case scenario

SG 5 – Perform Non-Functional Test Execution

SP 5.1 Execute non-functional test casesSP 5.2 Report non-functional test incidentsSP 5.3 Write test log

-Execute planned NF tests-Report NF test issues and track till closure-Capture test outcome


3.5 Peer Reviews


SG 1 – Establish a Peer Review Approach

SP 1.1 Identify work products to be reviewed SP 1.2 Define Peer Review Criteria

-Agree upon work products which will go through peer review- Expectation defined for peer review

SG 2 – Perform Peer Reviews SP 2.1 Conduct peer reviewsSP 2.2 Testers review test basis documentsSP 2.3 Analyze peer review data

-Peer review happens as per plan- Test engineers are part work product reviews which impacts their deliverables-Analyze peer review outcome and improve organization checklist



What you can see in L3 organizations?

• There is an established test organization

• Testing is integrated to SDLC

• Test plan is developed, tracked and controlled (Integrated with project plan)

• Key work products go through peer reviews

• Organization review checklist for all documents evolve

• Test engineers drive test process improvement

• Users/clients attend milestone meeting

• User/clients support in developing usability test plans


Thank YouEmail ID: [email protected]

tmmi implementation guideline

Technology