Download - Risk-based Testing
![Page 1: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/1.jpg)
ConfidentialPA12013-09-271
Risk-based TestingThe Devil is in the Details
![Page 2: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/2.jpg)
ConfidentialPA12013-09-272
Introduction• Johan Hoberg
• Sony Mobile▫~10 years▫Tester, Test Team Leader, Test Architect
• This is a presentation of my experiences of risk-based testing
![Page 3: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/3.jpg)
ConfidentialPA12013-09-273
Definition•Risk-based testing (RBT) is a type of
software testing that prioritizes the tests of features and functions based on the risk of their failure - a function of their importance and likelihood or impact of failure. (http://en.wikipedia.org/wiki/Risk-based_testing)
![Page 4: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/4.jpg)
ConfidentialPA12013-09-274
The Devil is in the Details• Refers to a catch or mysterious element hidden in
the details
• Derives from “God is in the Details”▫ Generally attributed to Gustave Flaubert (1821–1880)
• It is easy to ask for efficient, risk-based testing
• It is much more difficult to actually do it well
![Page 5: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/5.jpg)
ConfidentialPA12013-09-275
System Under Test
Android Jellybean Customer A
Android Gingerbread Generic
Android Gingerbread Customer A
Android Gingerbread Customer B
Configuration 1
Configuration 2
Configuration 3
Product X
Product Y
Android Jellybean Customer B
Android Jellybean Generic
![Page 6: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/6.jpg)
ConfidentialPA12013-09-276
System Under Test
![Page 7: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/7.jpg)
ConfidentialPA12013-09-277
Changes in the System•There is a change introduced somewhere
in the system
•Could be a bug fix, a change request, a new feature, refactoring, etc.
![Page 8: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/8.jpg)
ConfidentialPA12013-09-278
Scope Selection• Do you run every available test on every possible
configuration of the system and verify all interoperability?
• A smart scope is necessary to make it containable
• We must be able to reuse test results between products, configuration, and branches
![Page 9: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/9.jpg)
ConfidentialPA12013-09-279
Scope Selection based on Risk
Change Introduced
System Impact
Risk-based Scope
![Page 10: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/10.jpg)
ConfidentialPA12013-09-2710
Complex vs. Complicated•Complicated
▫Opposite of simple▫Containing intricately combined or involved parts
•Complex▫Opposite of predictable?▫This adjective means having parts so
interconnected as to make the whole perplexing
![Page 11: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/11.jpg)
ConfidentialPA12013-09-2711
Complex System•A complex system is difficult to predict
•It is therefore difficult to plan which actions to take to mitigate different outcomes
•It is difficult to plan which tests to run to cover all potentially relevant defects
![Page 12: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/12.jpg)
ConfidentialPA12013-09-2712
Reduce complexity
Reduce Complexity
Change
Complex System
Scope Selection
Very difficult!
![Page 13: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/13.jpg)
ConfidentialPA12013-09-2713
Example•A change is introduced into the Display Driver
component – What do I need to test?▫Camera?▫Interoperability with TV and similar devices?▫Multimedia
View pictures, video etc.?▫All UI testing?▫Other parts of the system?
![Page 14: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/14.jpg)
ConfidentialPA12013-09-2714
Reduce Complexity•Risk Analysis•Automated Test Framework•Early Exploratory Testing
![Page 15: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/15.jpg)
ConfidentialPA12013-09-2715
Risk Analysis – Input DataFeature & Hardware
Delta
Historical Data
Code Changes
System Dependencies & Architecture
•Easy to obtain•Might give us an indication on where to start looking, but can often give false confidence in scope selection, especially when it comes to E2E system test
•Hard to obtain
![Page 16: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/16.jpg)
ConfidentialPA12013-09-2716
Risk Analysis – Risk Model
Risk Identificatio
nProbability Consequenc
eMitigation
Plan
Severity (S)Occurrence (O)Detection (D)
Risk Priority Number (RPN) = S*O*D
![Page 17: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/17.jpg)
ConfidentialPA12013-09-2717
Risk Analysis - Complex vs. Complicated
Risk analysis of whole system is complex
Risk analysis of sufficiently small part is more complicated than complex
![Page 18: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/18.jpg)
ConfidentialPA12013-09-2718
Risk Analysis Database• If each function performs risk analysis on
their components and enters risks in a database this information could be used on system level to get better understanding of impact of specific changes
•Requires a uniform way of handing risk information
![Page 19: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/19.jpg)
ConfidentialPA12013-09-2719
Problems?• We can get a lot of data that will help us select a
scope – but ultimately we will introduce a lot of risk if we do not know how the system is designed, and how different components depend on each other
• To get a complete dependency map of the whole system requires not only a good risk-handling infrastructure, but a common effort from a lot of people
![Page 20: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/20.jpg)
ConfidentialPA12013-09-2720
Dependency Map Example
Driver
Library
Application Framework
Application Framework
Application
Application
Application
Application
Application
![Page 21: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/21.jpg)
ConfidentialPA12013-09-2721
Automated Test Framework
Change + Basic Risk Analysis
Scope Setting Test
Execution
Automated Test
Execution
Test Results Analysis
![Page 22: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/22.jpg)
ConfidentialPA12013-09-2722
Automated Test Framework
ChangeAutomated
Tests ExecutedFailed Tests Investigated
Change in component mapped towards failed
tests in other components
Better understanding of
System Dependencies
![Page 23: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/23.jpg)
ConfidentialPA12013-09-2723
Self-learning System•Over time you will have more and more
information about how different changes impact the system
•Map code changes to failed tests•This impact map can further help you
set an efficient scope
![Page 24: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/24.jpg)
ConfidentialPA12013-09-2724
Continuous Integration•The more code changes we have and the
more isolated these code changes are, the easier it is to create the impact map
• If you only have one integration per month, with thousands of lines of code changed in different parts of the system, it will be difficult to draw conclusions
![Page 25: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/25.jpg)
ConfidentialPA12013-09-2725
Problems?•The information value you receive is
dependant on how good the tests are•Dependency graph grows over time – in
the beginning there is very little data
![Page 26: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/26.jpg)
ConfidentialPA12013-09-2726
Early Exploratory Testing
Change + Basic Risk Analysis
Scope Setting Test
Execution
Exploratory Testing
Test Results Analysis
![Page 27: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/27.jpg)
ConfidentialPA12013-09-2727
Codifying Experience•How do you transfer the knowledge
gained from one tester to the other?•The understanding about impact and
dependencies we gain from performing exploratory testing must be stored in a way which makes it accessible to other testers for their risk-based scope selections
![Page 28: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/28.jpg)
ConfidentialPA12013-09-2728
Manual Dependency Map•After each exploratory session it is
possible to add information to a dependency map, where you indicate which changes were made, and what impact those changes had
•It needs to be done in a way which is quick and easy, otherwise it will not be done at all
![Page 29: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/29.jpg)
ConfidentialPA12013-09-2729
Problems?•Additional time might be needed before
testers get experience with the new process
•It will be more difficult to time plan activity before early exploratory testing is done – this can cause problems with test/project leaders and managers
•It requires skilled testers
![Page 30: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/30.jpg)
ConfidentialPA12013-09-2730
Final thoughts•Every risk-based scope introduces risk –
our goal is to eliminate any unnecessary risk
•Without skilled testers to write good automated tests or perform exploratory testing it becomes very difficult without a very intricate risk/impact framework
![Page 31: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/31.jpg)
ConfidentialPA12013-09-2731
Conclusion•It is very difficult to perform
risk-based testing for a large, complex system
•It is very easy to ask someone else to reduce their test scope by introducing risk-based testing
![Page 32: Risk-based Testing](https://reader035.vdocument.in/reader035/viewer/2022062319/554a06bab4c905e56c8b56fb/html5/thumbnails/32.jpg)
ConfidentialPA12013-09-2732
Contact•Email
•Slideshare▫www.slideshare.net/JohanHoberg