better managed outsourced software testing
TRANSCRIPT
![Page 1: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/1.jpg)
Better Outsourced TestingCreating and managing agreements with offshore vendors
![Page 2: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/2.jpg)
Introduction
This presentation will highlight three key failure points to outsourced testing:
1.misunderstanding of the cost of quality;
2.misalignment of contractual arrangements to the required testing/delivery outcomes; and
3.lack of effective metrics.
![Page 3: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/3.jpg)
Traditional outsourcing agreements..
![Page 4: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/4.jpg)
..focus on the test execution statistics and not quality as an objective.
![Page 5: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/5.jpg)
Is there a solution?
It is important to understand these issues can be mitigated and rectified by examining current testing capability and improving the testing processes, not by just policing the fine print in the outsourcing contract.
![Page 6: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/6.jpg)
Defining the problem
![Page 7: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/7.jpg)
Symptoms
Along with the typical less than optimal outcomes (e.g. poor quality, cost overruns and late delivery), other potential symptoms of poorly managed outsourced testing include:
![Page 8: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/8.jpg)
● poor or missing test planning documentation;
![Page 9: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/9.jpg)
● minimal test results;
![Page 10: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/10.jpg)
● deployment issues to test environments;
![Page 11: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/11.jpg)
● reactive focus on testing;
![Page 12: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/12.jpg)
● high severity issues upon deployment;
![Page 13: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/13.jpg)
● increases over time to the length of the test phase; and
![Page 14: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/14.jpg)
● little to none build or test automation.
![Page 15: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/15.jpg)
Organisations should “Understand that testing is required to verify and validate that a business objective is being realised... throughout the life cycle, not only at a user acceptance phase.
Therefore, clear functional and non-functional acceptance criteria should be defined for each phase of the life cycle.”
Gartner: ID:G00225707, ID:G00209445
![Page 16: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/16.jpg)
Misunderstanding Cost of QualityDifferent outcomes from a unit cost agreement and one where total cost of quality is an objective
Unit Cost vs. Total Cost of Quality
![Page 17: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/17.jpg)
Misaligned contractual arrangementsOften the commercial agreement is the only governance provided to measure, manage, report and maintain the output of the outsourced testing provider.
Different Outsourced Engagement Models
![Page 18: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/18.jpg)
The true value of an outsource provider is meeting the client’s delivery need with the capability (and ability) to deliver
![Page 19: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/19.jpg)
Lack of Effective Metrics
If an outsource testing provider is bound by unit cost measures like total defects found, the onus is on finding the maximum number of defects (regardless of quality of defects, analysis, triage, allocation, etc.) in the allotted period of time.
![Page 20: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/20.jpg)
Common Results
Typically, the outcome will be an unquantified list of unqualified defects, test results with a high failure or blockage rate, and no working solution.The outsource provider may then be rewarded with more resources to come up with another outcome (reduce the number of defects or pass tests), giving the provider more revenue and leaving the client with cost and time overruns.
![Page 21: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/21.jpg)
Organisations “have limited internal baseline metrics regarding quality and efficiency of their testing capabilities.”
Gartner: ID:G00225707, ID:G00209445
![Page 22: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/22.jpg)
What’s the solution?
![Page 23: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/23.jpg)
1: A Capability Review
Assessing the current testing capability may be as simple as reviewing past test results against estimated and planned outcomes. It could also involve a professional external organisation performing a capability review and auditing policies, processes, procedures and skills.
![Page 24: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/24.jpg)
What is a Capability Review
This review can be used to recommend improvements to specific areas. These could include a new or updated test strategy (or master test plan), to ensure the alignment to desired outcomes, the introduction of metrics to align to KPIs in the outsource contract, and defining effective (light touch) governance around roles and responsibilities to reach the desired outcomes.
Capability Review
![Page 25: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/25.jpg)
How and What to Measure?
relative test effort;
test coverage of requirements;
percentage of tests automated;
defects remaining upon completion, and
defect analysis.
These metrics should be holistically called the cost of quality.
![Page 26: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/26.jpg)
The cost of defects increases throughout the testing lifecycle and the longer a defect stays within the process, the more costly it is to fix.
![Page 27: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/27.jpg)
Cost of DefectsTo avoid cost overruns, consider using defect classification metrics to more easily identify trends in your test process
Cost of Fixing Software Defects
![Page 28: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/28.jpg)
To avoid cost overruns, consider using defect classification metrics to more easily identify trends in your test process.
![Page 29: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/29.jpg)
Defect reporting vs. defect analysis
Clients may already have effective defect reporting which can be invaluable for providing a clear view of the current status of quality. The move to defect analysis begins with reporting as an input, combined with additional quantitative information to allow analysis of what is called phase effectiveness.
![Page 30: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/30.jpg)
There are two key metrics that assist with understanding trends in quality:
![Page 31: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/31.jpg)
(PCE) Phase Containment Effectiveness
PCE is the ratio of faults captured in a phase and signifies how effective the testing process is at preventing these defects in the first place.
![Page 32: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/32.jpg)
(PCE) Phase Containment Effectiveness
In Figure 4, the PCE is calculated as follows:
Requirements captured 10 of 22 defects = 45%
Design contained seven of 22 defects = 32%
Build contained three of 22 defects = 14%
Test contained two of 22 defects = 9%
Phase Containment Effectiveness
![Page 33: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/33.jpg)
(PSE) Phase Screening Effectiveness
PSE is the ratio of previously missed (not contained) defects being captured in each phase.
For the same example, the PSE is calculated as:
Requirements missed 12 of 22 defects
Design screened seven of these 12 defects = 58%Build screened three of these 5 defects = 60%
Test screened two of the remaining two defects = 100%
![Page 34: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/34.jpg)
Create or update a testing framework, which will fully align these metrics to the total cost of quality.
![Page 35: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/35.jpg)
Organisations failing to fully understand their outsourced contracts and not including effective metrics and role accountabilities to monitor these often see cost overruns and project delays.
![Page 36: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/36.jpg)
KPIs drive behaviour
Organisations will be left with statistics to prove the KPI was met, but a less than optimal quality outcome
![Page 37: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/37.jpg)
3 Simple Steps to Rectify
Outsourced testing can be successful.
It can speed up the test execution, improve quality and reduce costs.
It is important to ensure the total cost of quality is understood, the quality outcomes are defined and aligned to the commercial agreement.
![Page 38: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/38.jpg)
1: Capability Review
Involve a professional, external organisation to perform a capability review
![Page 39: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/39.jpg)
2: Outsourced Agreement Review
Recommend improvements or enhancements to the organisation’s governance of the commercial arrangement and the expected testing outcomes
![Page 40: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/40.jpg)
3: Improved Metrics
Together with changes to the metrics for both the outsourced provider and the client organisation should be used to report against the testing outcomes
![Page 41: Better Managed Outsourced Software Testing](https://reader036.vdocument.in/reader036/viewer/2022062311/58efa95d1a28ab2e188b45ad/html5/thumbnails/41.jpg)
Nathan Shearer - Want to work with me? rapidexecutive.com/coaching
Like This?Feel Free to Share it with someone who could benefit → Connect with me on LinkedInConnect with me on Twitter