PowerPoint Presentation

Statistical Process Control & MetricsSuggestions to alter perceptions of the value of testing.

Mike Jarred & Ilca Moussavou

Slide # IDBS 20121Common challenges (perception & value) for Test Managers to solveChallenge ObjectiveValue of testing not universally accepted

(the NEED for testing is often accepted)

Demonstrate and gain recognition of the value of the testing groupMake test group the agent for change in the companyTesting perceived as a bottleneck

Alter perception of testing as a bottleneck to a barometer of project health

More FTE Required than budget would accommodateDemonstrate how rework levels consuming test resources improve quality to reduce rework. Divorce rework from testing as a cost (guilty by association)Isolated objective targets across departments make it hard to demonstrate value / drive strategic initiativesProvide information to Executive Team so they can set cross departmental targets

Testing process improvement driven by practitioners, but not necessarily aligned to business objectivesEnsure test improvements support business goals

Quality viewed subjectively, with different interpretation by different stakeholders. Only quantitative measure is defect count.Remove the subjectivity of quality and work towards a measurable quality definition. Show value add of test by driving quality initiatives, not just measuring it. Slide # IDBS 20122

Slide # IDBS 20123A problem sharedWorld Quality Report 2012 Sogeti / HP (global survey of more than 1500 CIOs, CFOs, IT and QA Directors) Less than 5% of firms regard their testers as best in class is it really that low, or perception?Testing needs to be a formalised step in the application lifecycle we still see instances of the testing function being considered as an afterthought or viewed as a roadblock preventing application deploymentSo, this is what our bosses are saying about usWhat can we do to address this perception this is their reality

Slide # IDBS 20124No silver bullet, or standard approach

Slide # IDBS 20125Demonstrate and gain recognition of the value of testing

If there is no measure of the protection / assurance testing provides to the business this inevitably raises a question of value

Often there are only limited existing measures; e.g. time in test per project, and defect data raised by projects.

Introduction of Test Effectiveness Metric (aka Defect Leakage, with a positive spin - if your figures are good!)

Test Effectiveness = defects found in test as percentage of defects found by test and customers

Demonstrate per product line the Test Effectiveness, with improvement over time

A communication process for this will be covered later in the slide deck

Gain advocates for example a Support Manager or Product ManagerThey feel the pain if test dont add valueThey understand the impact of good testing on their departments, the organisation and customers

Slide # IDBS 20126Example Test Effectiveness timeline

%TimeSlide # IDBS 20127Changing the perception of testIntroduce daily reporting that shows the trends of Quality & Progress over time of the project constraints to test progress easily demonstrable. Increasing Visibility & Predictability = value add!Start reporting these metrics only on the high risk / critical projects Once the opportunity cost (e.g. less projects being tested) is understood by Executive, testing no longer seen as a bottleneck, but the solution is to the left of testing in the lifecycle

Testing can be perceived as a bottleneckCapture time to demonstrate how much effort is applied to rework (or, WASTE!)Introduction of Initial Build Quality (IBQ) (% of tests that pass FIRST time they are run, for new functionality and regression)IBQ = 50 tests executed, 25 pass, 25 fail = 50% IBQ measure of development process.

Slide # IDBS 20128Daily Test Report illustrative trend information

Slide # IDBS 20129Demonstrate rework levels impact on available test resource: illustrative slippage due to rework

Effort (Hours)

DateSlide # IDBS 201210Raising value awareness Internal Conference? Examine possibility of promoting ideas internally, e.g. a conference day

Invite VPs, Sales, Marketing, Product Management, Project Management, Development, Support (any key stakeholders and advocates who provide internal independent references for testing)

Make testers feel valued, and grow profile of the test group.

Market this internally the same way an external events company would promote a conference, add external speakers to add credibility / gravitas

This can significantly raise the profile of testing

Demonstrate increased test effectiveness with an increase of workload (if this is a reality it often is)

Demonstrate IBQ, discuss the impact on test resourcing and waste

Discuss, amongst other things, how testing could support quality further given understanding of customers needs / want / perception

Slide # IDBS 201211Inform Executive Team so cross departmental targets can be set Create awareness that departmental improvements need to support other departments, and relate to strategic business goals

Propose, and take ownership of running a Metrics Governance Board.Snr. Managers representing IT & Business a senior stakeholder owns the meeting (but testing run it)Executive summary used by senior management in leadership team meeting with CEO

Reporting for all Products - processes needed base liningUnit Test Coverage & IBQ, Overall Quality (Development Process Measures)Test Effectiveness (Test Process Measure) Support metrics support SLA achievement

Once Performance baselines are understood, targets for improvements can be set.

Slide # IDBS 201212Statistical Process Control (SPC)Simple approach to quality control and process improvement involving statistical data analysisSPC consists of: - Measuring of a process (using objective data) - Eliminating variation due to assignable causes - Regular monitoring of a process - Improving the process to its best target valueTool Used: Control ChartTotal Variation = Common Cause Variation + Special Cause Variation

Slide # IDBS 201213Statistical Process Control Day to day example Special Cause VariationE.g.: Heavy Traffic, Accident on the road, Woke up lateDaysJourney TimeMonday45 minutesTuesday50 minutesWednesday47 minutesThursday120 minutesFriday49 minutesSlide # IDBS 201214Statistical Process Control Control Chart

Slide # IDBS 201215Ensure SDLC improvements support business goals Root Cause Analysis of control limit breaches drives change to process to reach departmental targets.

Slide # IDBS 201216Statistical Process Control - Benefitsa) Continuous Process Improvement

b) Process Baselines

c) Early visibility and reaction

d) Quantitative Management Decision Making

e) Economical ValueSlide # IDBS 201217Does this make me a bad parent?Pocket money is earned , and lost , in 5 pence increments depending on behaviour

Sunday is count-up day - and root cause analysis!Slide # IDBS 201218Building Trust - Credibility of MetricsMetrics provided by SPC need credibility

Show a Governance Board the following;Approach taken in capturing data, including the source of data

Historic data, (up to 4 years old) that not only provide the baseline but show outliers (events) in the corporate memory they recognise, so future outliers will be trusted.

The data is usually available in Quality Centre & other management toolsets it is simply a case of extracting it and presenting the information.

Slide # IDBS 201219Target SettingInitial targets set, across the SDLC

Targets set using historical data, with attention to the previous 6 9 months

As the targets are set using detailed knowledge, the targets are seen as fair and reasonable

Target setting should involve the customer of the process; e.g. IBQ is a measure of development build Test can set expectations of acceptable quality as a development target. Support can set the target for test effectiveness

Slide # IDBS 201220The vision - transition of testing to agents of changeTest can have a credible voice at the top table, with authority to challenge conventional thinking

Process improvements across the organisation can be driven through testing and the information the function provides

Test Group can position itself positioned as the trusted advisor - seen by execs as the advocates in improvements to the SDLC

Metrics Governance Board can trigger improvement initiatives to Increase predictability of project outcomesImprove quality of software into testMove towards defect prevention culture

Our conference output has seen major investment in performance, security & business analysis to improve usability and quality of products.

Slide # IDBS 201221Success FactorsStandardised Testing Methodology, consistently applied

Standardised testing tools and SDLC tools to capture Data surrounding your process. Ability to extract the data from QC database.

Establish responsibilities for data capture, presentation & utilisation.

Change the culture use advocates / champions outside your group help

Good communication (top / down / sideways). Create feedback loops

Access to a mathematician if you dont have an Ilca - tools like Minitab for SPC

Ensure sponsorship work out what is important to your sponsor and how this would help them achieve their goals. Youll probably get permission. If not, do it anyway - you may be seen as innovative

Keep it simple Senior Execs dont have the time for detail, or necessarily understand the detail

Stress metrics are used for process improvement not individual performance management (there is no quicker way to derail any inertia!)

Slide # IDBS 201222Closing thoughtsYou already have lots of data use it to sell and tell a storyEverybody has a process, even agile teams! its about measuring what you feel is significant in your process to see if it can be improved Ensure the information is credibleBe aware that metrics can drive behaviour (not always favourably)

Use industry information and compare blow you own trumpet when you smell a success story (e.g. WQR only 5% companies achieve 100% ROI on automation) build your case, no one will be able to confidently challenge (you are the experts!)

Slide # IDBS 2012Blues51336.0