Key Principles for Efficient Test ReportingProviding the right information to the right stakeholders

Purpose

• This document outlines the key principles of efficient test reporting

• This is a high level presentation about test reporting that must be implemented differently for every context and stakeholder

• Following these key principles will give a good base for setting up efficient test reporting

Key Principles for Efficient Test Reporting

Confidence in Measurements

Risk / MitigationQualitative / Quantitative

Traceability in Results

Stakeholder Expectations

Key Principle: Stakeholder Expectations

• What info is suitable for the intended stakeholders?– What info do they want?– What info do they actually need?– What decisions are being made based on the information?

• How do the stakeholders need the information reported to easily be able to use it?

• What granularity do they need? High level overview or technical details?

• What data to they need to pass along in their consolidated reports for others to make decisions on?

• Answers the question: What, how and why do we report?

Key Principle: Risk / Mitigation

• Why did we test at all?

• What were the goals of the test activities?

• What product or project risks initiated the test activity/activities that the report covers?

• Did the testing mitigate the problems/risks?

• How can remaining risks be mitigated?

• Answers the question: Why did we test?

Key Principle: Confidence in Measurements

• It must be clear what the scope for the test activities were on different abstraction levels so that stakeholders can get a good overview but also gradually go into more detail

• The stakeholder must be able to understand how confident we are in our test results

• If status is green, how good are the chances that it will stay green, and how much risk is there that the status will change over time

• Are the test results reliable as decision material

• Answers the question: What did we test, and what didn’t we test?

Key Principle: Qualitative / Quantitative

• We need both data and an analysis of that data from a professional in the report

• The analysis leads to a statement about the product, and this statement needs to be backed by empirical data

• The empirical data needs context that only a professional can put it in – the product statement is this context

• One cannot be reported without the other

• If focus should be on the data or the analysis depends on the stakeholders

• Answers the question: How do we report?

Key Principle: Traceability in Results

• The results reported in the Test Report must be reproducible by stakeholders

• There must be traceability between the results and how it was executed

• Some or all of the following variables must be covered to create traceability

– Which tools were used– How was the test environment configured– Work process – Links to test cases – Links to defects found– Links to logs and other detailed test artifacts

• Answers the question: How did we test?

Summary

• Using these key principles will not automatically give good test reporting, but not following these key principles will most likely give insufficient test reporting

• Know your stakeholders, and understand the key principles of test reporting