creating sanity amidst the test methodology madness - webinar presentation
DESCRIPTION
This webinar was presented by MindTree Ltd. and Forrester Research Inc. on the 14th of June 2011.TRANSCRIPT
© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 1 © 2011 MindTree Limited CONFIDENTIAL: For limited circulation only
Creating Sanity Amidst Test
Methodology Madness
Right-sizing your test methodologies
© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 2
Agenda
hen hat ow
Identify the
symptoms
Key aspects
to right-size
MindTree
Tools &
Frameworks
Processes should work for you. You should not work for processes.
Rightsizing?
© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 3 © 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 3
Poll Question #1
● Are your processes taking too much effort and time?
● Yes
● No
● Not able to measure
© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 4
When to Rightsize?
Acknowledge the need. Don’t wait until you loose your sleep.
• Test processes taking too much effort and time?
• Metrics not helping you take decisions?
• Not able to justify the investments in testing?
• Test process not helping you identify the critical
defects early?
• Difficulty in quantifying the quality in your test
processes?
hen
Identify the
symptoms
© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 5
What to Rightsize?
Tailor your processes for right fit.
hat
Key aspects
to right-size
• Quality goals / objectives
• Software life cycle methodologies
• Engagement models
• Governance mechanism (metrics,
reviews)
• Tools and frameworks
© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 6 © 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 6
Poll Question # 2
● Do you know what metrics should you be tracking for your testing
needs?
● Yes
● No
● I do, but not sure if they are exhaustive
© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 7
Test Metrics Analysis and Decisions (TMAD) Model
Metrics
Based
Governance
• Business goals
• Quality goals
• Governance
• Metrics, reviews,
baselines
• Forecast
• Risks, best practices
• Corrective & preventive
actions
Test Metrics Analysis and Decisions (TMAD) Model
Statistical
Models
Historical
Data
Org / Industry
Baselines
Enablers
ow
MindTree
Tools &
Frameworks
© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 8
Test Metrics Analysis and Decisions (TMAD) Model
Proposed Metrics Goal Associated Queries (RCA) Undesired Behaviour
(Risks)
Defect Containment Effectiveness
1. Improve testing quality
2. Prevent defect leakage
1. Was the criteria to classify defect detection phase clear from a leakage perspective?
2. Are there clear entry and exit criteria between phases 3. Was the team's knowledge adequate to uncover
leaked defects? 4. Was the scope of testing for each phase clear? 5. Do we have requirement traceability matrix?
1. Blame game 2. Manipulation of defect
classification criteria
Decision Support (Scope) Corrective Actions
Raw Data Requirements Suggestions (feedback)
1. Improve test case review effectiveness
2. Improve process compliance 3. Improve product and test design
knowledge
1. Clear entry and exit criteria between phases
2. Process adherence 3. Effective knowledge management 4. Update test suite 5. Get test strategy reviewed by
customers
1. Defects detected in test phase
2. Leaked defects
Do RCA for every release and identify preventive actions
Metrics Analysis
Metrics Decision Table
© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 9 © 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 9
Test Quality Index (TQI) Model
● Bivalent questionnaire; Y or N
responses
● “Productivity per tester” vs.
“quality of deployed resources
and resource deployment”
● Contextual metrics vs.
comprehensive score; across
30 process areas
● Scoring based on self
assessment
1 Identify
areas of
improvement
2 Customize
the scored
checklist 3 Improve
scores with
each
deliverable
Winner of the Testing Leadership Award at QAI Software Testing Conference
© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 10 © 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 10
Poll Question # 3
● Do you have an integrated framework which helps you rightsize the
test processes and methodologies?
● Yes
● No
● We have point solutions, but not an integrated solution
© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 11
MindTest™ - Integrated Test Methodology
Feedback
Analysis Audits
Test Estimates
TQI TMAD
Test Management
TestLink
Test Measurement and
Metrics - ROI
Test Dashboard
Lifecycle Processes
ProcessNet
Knowledge
Management
ProjectSpace
Defect Management
Radar
Tools Technologies Domain Type of Testing Stage of Testing
Customization
Processes Templates Guidelines Checklists Best Practices
Continuous
Processes
Metrics &
Measurement
Lifecycle
Processes
© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 12
• Executed in-depth analysis to understand
• Quality objectives and alignment between test process
and application's business goals
• Used Test Metrics Analysis and Decisions (TMAD) Model
and Test Quality Index (TQI) to come up with
appropriate metrics and processes
• Collected key metrics as governance enablers to measure and
drive decisions based on the health of the projects
MindTree Solution
The Business Challenge
The customer is the world's leading specialist in air transport
communications and information technology (IT) solutions.
• 90% coverage on test cases executed, a significant
improvement over the earlier metric
• 100% achievement on acceptable open defects for severity 1
and 2
• Customer overhead of test case reviews reduced by 60%
• Constant review of the test case execution helped us to
evaluate and report how far/near is the application from the
acceptance criteria at different phases of testing
• Enabled development team to prioritize the defects
that needs to be fixed for the immediate build
• Customer is now able to ascertain the quality of the
application
• Reduced risks and enabled effective decision making through
test dashboard
• No testing guidelines
• Inadequate process
• Lack of measures for impact of testing
• No single snapshot of engagement level governance
The Customer Business Impact
“MindTree's expertise in metrics is paramount in managing
the engagement and seeing if process changes have made the
hoped for improvement and if the application is ready for
release. The metrics are also a great way to communicate
progress across all the groups in this engagement.”
Senior Manager, Quality Integration
Case Study
© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 13 © 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 13
In Short…
You don’t have to search for clues in the
labyrinth
Identify the need
Evaluate the key aspects
Optimize the process
Measure and improve
© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 14
MindTree Testing Services Overview
Testing Size and
Focus
• Dedicated unit of 2200+
specialists focused on
testing
• Contributes 25% of the
MindTree’s global
business
Testing Depth and
Breadth
• Tested over 2000
applications & products
from chip to cloud
• Services ranging from
point solutions to long
term CoEs
• Domain specific testing
services and solutions
Testing Process and
Delivery Excellence
• Testing pillars of
excellence - MindTest™,
TestLabs,
TestingAcademy
• Virtual practices and
CoEs for effective
delivery
• Part of ISO working
group in defining new
testing standard – ISO /
IEC 29119
• President of India’s
leading testing forum -
STeP-IN
Thought Leadership
1500+ 95% 700+ 20+
delivering PREDICTABLE QUALITY through INDEPENDENT TESTING
© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 15
ARE YOU TRACKING THE RIGHT TEST METRICS
FOR YOUR BUSINESS DECISIONS?
Complimentary Workshop at your Doorstep
REGISTER NOW
or email to [email protected]
www.mindtree.com/events/testingwebinar/index.html
MindTree Workshop
© 2011 MindTree Limited CONFIDENTIAL: For limited circulation only Slide 16 © 2011 MindTree Limited CONFIDENTIAL: For limited circulation only
Sylvia Veeraraghavan
www.mindtree.com