Non-Functional Testing 1UKTMF27th January 2010

Non-Functional Testing Non-Functional Testing   • Why is this so often done badly or not done at all?

• Can we build a Test Managers toolkit to help?• What would it contain?

Paul Godsafe

Non-Functional Testing 2UKTMF27th January 2010

Non-Functional TestingNon-Functional Testing• Reliability - The capability of software to maintain its level of performance under stated

conditions for a stated period of time. – Maturity, Recoverability, Fault Tolerance

• Usability - The effort needed for use, and the individual assessment of such use, by a stated or implied set of users.

– Learnability, Understandability, Operability

• Efficiency - The relationship between the level of performance of the software and the amount of resources used, under stated conditions.

– Time Behaviour, Resource Behaviour

• Maintainability - The effort needed to make specified modifications. – Stability, Analyzability, Changeability, Testability

• Portability - The ability of software to be transferred from one environment to another. – Installability, Replaceability, Adaptability, Conformance (e.g. to a particular database standard)

Ref. ISO 9126

Non-Functional Testing 3UKTMF27th January 2010

Why is Non-Functional Testing so Why is Non-Functional Testing so often done badly or not at all?often done badly or not at all?

• Need not clearly defined or understood• Creates problems, not solutions

– Need to define requirements– Where do you start (or end)?– Often hard to understand and/or interpret results

• Expensive to do– People, kit, software, time

• Risk/reward is hard to quantify in advance– Are you an optimist or pessimist?

• Solutions rarely available on demand– Cost, lead times, reliability, access to skills…….– Intermittent need

Non-Functional Testing 4UKTMF27th January 2010

Oh what to do,Oh what to do, what to dooooo?what to dooooo?

Non-Functional Testing 5UKTMF27th January 2010

Well, we could………Well, we could………

• Maintain an idealistic view

• Just complain

• Do nothing and accept the status quo

• Talk to other TMs and agree that nobody loves us

• Develop some pragmatic solutions based on……….

Non-Functional Testing 6UKTMF27th January 2010

Genuine Need, Realistic Demand and Genuine Need, Realistic Demand and Access to solutions (tools/skills/services)Access to solutions (tools/skills/services)

• Attitude to risk• Commercial/operational drivers• Attitude towards testing• Centralised/devolved organisation• What can be realistically delivered

• Attitude may be influenced by:– Past experience– The way you provide advice, guidance & solutions

Non-Functional Testing 7UKTMF27th January 2010

So can we build a Toolkit to Help?So can we build a Toolkit to Help?

Possibly but……….

• A toolkit can only be used on a defined problem. Are you looking for, analysing or solving non-functional defects?

In addition you may need:• A strategic objective to drive the scale/scope• Demand to use your toolkit

– Where will the non-functional requirements come from?• An interface to development projects• A strategy for delivering your toolkit - self-service, internal or 3rd

party– Scalability and demand management– A delivery policy/mandate

• A means of payment

Non-Functional Testing 8UKTMF27th January 2010

Does your toolkit address the Does your toolkit address the causes of inertia….causes of inertia….

• Does it define need and improve understanding?

• Does it provide a complete solution?• Is it cost-effective?• Does it deliver value-add?• Is it available when needed?

• Virtual NFT – the power of suggestion

Non-Functional Testing 9UKTMF27th January 2010

The End of the BeginningThe End of the Beginning