Key Principles for a Generic Test Organization

Purpose

• This presentation outlines a number of key principles that could be

applicable for any test organization

• By applying these key principles to a test organization it becomes

more efficient and effective

• Not everything is applicable in every context, but the key principles

should be generic in nature and usable under most circumstances

Key Principles

Testing as a Service

Smart Automation

Useful Metrics

Risk-based Testing

Customer Focused

Reporting

Changing Requirements

Context Dependent

Test Methods

Empowering Tools

Right Competence

Key Principle: Testing as a Service

• A tester is not a quality police – clients make decisions based on a

service the tester provides [1]

• A test organization should provide an understandable and simple

service catalogue of how they can help their clients get the

information they need to make the correct decisions

• A test organization should be able to support their clients in selecting

what services are most suitable for their specific needs

• A test organization should always be able to show and explain their

value

Key Principle: Smart Automation

• Smart test automation is not automating a set of manual tests [2]

• Smart automation is designing a good automated test scope from the

beginning

• Manual tests are not designed to be automated – create automated

tests that are suitable for automation, don’t force automation on

something that was designed to be manual

• Always have a return-on-investment analysis, and consider

maintenance, portability and re-usability costs

• Drive testability in the code so that automation becomes a natural

part of the development process

Key Principle: Useful Metrics

• Not everything that counts can be measured. Not everything that can

be measured counts. [3]

• Metrics should be used to pinpoint where an analysis or audit must

be conducted, not as hard evidence

• Always consider what kind of behavior a certain metric will promote

before communicating it

• Never be afraid to measure, but understand that metrics never give a

complete picture

Key Principle: Risk-based Testing

• All testing should be based on a proper risk analysis [4]

• Clients decide what they want, but a tester must be able to explain

why a certain scope is more or less effective for the clients needs and

acceptable risk levels

• Use as much empirical data as possible for these risk analysis, don’t

just go on a gut feeling – quantify that feeling when possible

• Inform the client of the costs of different levels of uncertainty and

risk; if the client wants everything tested and no risk it will cost an

indefinite amount, if the clients accepts any risk, then nothing is

tested, and the cost is zero

Key Principle: Customer Focused Reporting

• Don’t ask the client what information he wants – ask what decisions

the clients intend to take based on the information

• Tailor the test and defects reports based on the clients needs;

qualitative and quantitative information can be equally powerful –

but always back up statements with facts

• A tester must communicate what confidence the client can have in

the results – just because no issues are found does not mean that the

software is bug free

• But always remember; “We don’t manage the project managers. We

don’t decide what information they have to give to the people they

report to.” [5]

Key Principle: Changing Requirements

• Don’t expect complete requirements up front

• When parameters change, the tester must adapt to meet new

stakeholder needs; be adaptable to changing requirements during a

project

• If requirements are needed and none are available, write down

expected behaviour and request stakeholder acknowledgement –

communication with stakeholders is key when requirements are

unclear

• Try to minimize unnecessary test artefacts – combining

requirements and test cases into one entity can be one way forward

Key Principle: Context Dependent Test

Methods & Approaches

• The most effective and efficient testing is context dependant [6]

• It is the tester’s job to find the most effective and efficient method for

each context

– Scripted testing - Exploratory testing

– Black box - Gray box - White box

– Manual - Automated

– Tool supported or not

– Boundary analysis -Equivalence partitioning -Fuzz testing - State transition testing

– Etc.

Key Principle: Empowering Tools

• A tester should spend a majority of the time testing – streamline

everything else with the help of tools [7]

• Risk-analysis tools help the tester perform more efficient and

effective risk analysis

• Integrated reporting tools help the tester file defects without

changing systems [7]

• Tools to support time efficient test plans [8]

• Record and playback framework for repeating tests [9]

Key Principle: Right Competence

• There are many different tester roles that each require a diverse set

of skills – one size does not fit all

• Understand what roles are needed in a specific organization and

secure that the testers have the skills needed to fill those roles

• Make sure that every tester has an idea of what their career might

look like in five years, based on their specific skillsets [10]

• Understand that it is not a numbers game – do not hire more testers

to solve a problem – secure that you have the right testers [7]

• More does not equal better – it can sometimes lead to more entropy

and chaos

Summary

• These key principles should be applicable for most test organizations,

even if there will always be some exceptions

• While they do not cover everything, they should cover the most

critical areas that most test organizations need to think about

• There is no best practice – but this is a good start to help a test

organization finding a best practice for their context

Reference

[1] Ten Things about Testing that should die: Egocentricity

http://scott-barber.blogspot.com/2011/11/10-things-about-testing-that-should-die.html

[2] Exploring Test Automation

http://angryweasel.com/blog/?p=412

[3] Albert Einstein qoute

http://www.famous-quotes.net/Quote.aspx?Not_everything_that_counts_can_be_measured

[4] Risk-based Testing

http://en.wikipedia.org/wiki/Risk-based_testing

[5] Contexts Differ

http://context-driven-testing.com/?p=38

[6] Context Driven Testing

http://context-driven-testing.com/

[7] Turning Quality on its Head

http://www.youtube.com/watch?v=cqwXUTjcabs

[8] The 10-minute Test Plan

http://googletesting.blogspot.com/2011/09/10-minute-test-plan.html

[9] RPF

http://googletesting.blogspot.com/2011/11/rpf-googles-record-playback-framework.html

[10] Career Paths using Tester Personas

http://www.stickyminds.com/s.asp?F=S16548_COL_2