risk-based testing

ConfidentialPA12013-09-271

Risk-based TestingThe Devil is in the Details


Introduction• Johan Hoberg

• Sony Mobile▫~10 years▫Tester, Test Team Leader, Test Architect

• This is a presentation of my experiences of risk-based testing


Definition•Risk-based testing (RBT) is a type of

software testing that prioritizes the tests of features and functions based on the risk of their failure - a function of their importance and likelihood or impact of failure. (http://en.wikipedia.org/wiki/Risk-based_testing)

http://en.wikipedia.org/wiki/Risk-based_testing


The Devil is in the Details• Refers to a catch or mysterious element hidden in

the details

• Derives from “God is in the Details”▫ Generally attributed to Gustave Flaubert (1821–1880)

• It is easy to ask for efficient, risk-based testing

• It is much more difficult to actually do it well


System Under Test

Android Jellybean Customer A

Android Gingerbread Generic

Android Gingerbread Customer A

Android Gingerbread Customer B

Configuration 1

Configuration 2

Configuration 3

Product X

Product Y

Android Jellybean Customer B

Android Jellybean Generic


System Under Test


Changes in the System•There is a change introduced somewhere

in the system

•Could be a bug fix, a change request, a new feature, refactoring, etc.


Scope Selection• Do you run every available test on every possible

configuration of the system and verify all interoperability?

• A smart scope is necessary to make it containable

• We must be able to reuse test results between products, configuration, and branches


Scope Selection based on Risk

Change Introduced

System Impact

Risk-based Scope


Complex vs. Complicated•Complicated

▫Opposite of simple▫Containing intricately combined or involved parts

•Complex▫Opposite of predictable?▫This adjective means having parts so

interconnected as to make the whole perplexing


Complex System•A complex system is difficult to predict

•It is therefore difficult to plan which actions to take to mitigate different outcomes

•It is difficult to plan which tests to run to cover all potentially relevant defects


Reduce complexity

Reduce Complexity

Change

Complex System

Scope Selection

Very difficult!


Example•A change is introduced into the Display Driver

component – What do I need to test?▫Camera?▫Interoperability with TV and similar devices?▫Multimedia

View pictures, video etc.?▫All UI testing?▫Other parts of the system?


Reduce Complexity•Risk Analysis•Automated Test Framework•Early Exploratory Testing


Risk Analysis – Input DataFeature & Hardware

Delta

Historical Data

Code Changes

System Dependencies & Architecture

•Easy to obtain•Might give us an indication on where to start looking, but can often give false confidence in scope selection, especially when it comes to E2E system test

•Hard to obtain


Risk Analysis – Risk Model

Risk Identificatio

nProbability Consequenc

eMitigation

Plan

Severity (S)Occurrence (O)Detection (D)

Risk Priority Number (RPN) = S*O*D


Risk Analysis - Complex vs. Complicated

Risk analysis of whole system is complex

Risk analysis of sufficiently small part is more complicated than complex


Risk Analysis Database• If each function performs risk analysis on

their components and enters risks in a database this information could be used on system level to get better understanding of impact of specific changes

•Requires a uniform way of handing risk information


Problems?• We can get a lot of data that will help us select a

scope – but ultimately we will introduce a lot of risk if we do not know how the system is designed, and how different components depend on each other

• To get a complete dependency map of the whole system requires not only a good risk-handling infrastructure, but a common effort from a lot of people


Dependency Map Example

Driver

Library

Application Framework

Application Framework

Application

Application

Application

Application

Application


Automated Test Framework

Change + Basic Risk Analysis

Scope Setting Test

Execution

Automated Test

Execution

Test Results Analysis


Automated Test Framework

ChangeAutomated

Tests ExecutedFailed Tests Investigated

Change in component mapped towards failed

tests in other components

Better understanding of

System Dependencies


Self-learning System•Over time you will have more and more

information about how different changes impact the system

•Map code changes to failed tests•This impact map can further help you

set an efficient scope


Continuous Integration•The more code changes we have and the

more isolated these code changes are, the easier it is to create the impact map

• If you only have one integration per month, with thousands of lines of code changed in different parts of the system, it will be difficult to draw conclusions


Problems?•The information value you receive is

dependant on how good the tests are•Dependency graph grows over time – in

the beginning there is very little data


Early Exploratory Testing

Change + Basic Risk Analysis

Scope Setting Test

Execution

Exploratory Testing

Test Results Analysis


Codifying Experience•How do you transfer the knowledge

gained from one tester to the other?•The understanding about impact and

dependencies we gain from performing exploratory testing must be stored in a way which makes it accessible to other testers for their risk-based scope selections


Manual Dependency Map•After each exploratory session it is

possible to add information to a dependency map, where you indicate which changes were made, and what impact those changes had

•It needs to be done in a way which is quick and easy, otherwise it will not be done at all


Problems?•Additional time might be needed before

testers get experience with the new process

•It will be more difficult to time plan activity before early exploratory testing is done – this can cause problems with test/project leaders and managers

•It requires skilled testers


Final thoughts•Every risk-based scope introduces risk –

our goal is to eliminate any unnecessary risk

•Without skilled testers to write good automated tests or perform exploratory testing it becomes very difficult without a very intricate risk/impact framework


Conclusion•It is very difficult to perform

risk-based testing for a large, complex system

•It is very easy to ask someone else to reduce their test scope by introducing risk-based testing


Contact•Email

▫[email protected]

•Slideshare▫www.slideshare.net/JohanHoberg

mailto:[email protected]

http://www.slideshare.net/JohanHoberg

risk-based testing

Technology