white_paper_rbt

Upload: spiker009

Post on 08-Apr-2018

219 views

Category:

Documents


0 download

TRANSCRIPT

  • 8/7/2019 White_Paper_RBT

    1/16

    A nanthakrishnan JArchitect, Sonata Software

    Sonata Software Limited

    www.sonata-software.com

    Risk-Based Testing:

    Implementation of Risk-BasedApproach for

    Quality & Cost Optimization

    Technical White Paper

    AuthorKalyanam Kannan

  • 8/7/2019 White_Paper_RBT

    2/16

    Technical White Paper www.sonata-software.com

    Risk-Based Testing 1 Sonata Software Limited

    STATEMENT OF CONFIDENTIALITY

    Information included in this document, in its entirety, is considered both confidential and proprietary to

    Sonata Software and may not be copied or disclosed to any other party without its prior written

    consent.

    All logos used in this document are registered trademarks of the respective organizations.

  • 8/7/2019 White_Paper_RBT

    3/16

    Technical White Paper www.sonata-software.com

    Risk-Based Testing 2 Sonata Software Limited

    Abstract

    As a practiced trend in IT projects, Testing is performed only towards the end of a project. Teams

    dedicate hours to test possible risks and flaws after the project is ready to run. As software testing at

    this level invites several last minute modifications that can cause discomfort, or sometimes even refute

    the very concept of the project, it has become the need of the hour to come up with a way to ensure

    detection and reduction of risks, at an early stage of the project. Risk-Based Testing, or RBT as referred

    to in this paper, is a procedure in software testing which is used to prioritize the development and

    execution of tests based upon the impact and likelihood of failure of the functionality or aspect being

    tested based on existing patterns of risk.

    Taking a cue from the age-old saying of Precaution is better than cure, RBT aims to find areas where

    possibility of risk or defect is most likely to occur. Through this testing technique, a software test

    engineer can now select tests based on risk even before the initiation of the project. Example, through

    software testing, one can detect 200 errors by testing 5000 defects. RBT on the other hand, enables the

    software tester to pick only 500 probable defects areas and conclude with 190 defects, thereby saving

    the effort and time of the software tester.

    This paper outlines the Risk-Based Testing approach and describes how Risk-Based Testing can positively

    impact the development life-cycle based on business-oriented factors, offering organizations an

    actionable plan for starting a Risk-Based Testing approach for projects.

    About the Author

    Kalyanam Kannan has been in the software industry and testing for the past 14 years and has managed

    testing projects using different engagement models, Currently, as a Practice Director in testing, he is

    responsible for controlling the quality of releases and delivery with optimum cost. He is also involved in

    providing testing solutions using the latest technology, tools and operating models, which enable

    projects to minimize cost to quality. His current areas of interest include Risk Based Testing, Test Driven

    Development and Open Source Based Testing.

    If you would like to interact with the author of this White Paper, feel free tocontact us.

    mailto:[email protected]:[email protected]:[email protected]:[email protected]
  • 8/7/2019 White_Paper_RBT

    4/16

    Technical White Paper www.sonata-software.com

    Risk-Based Testing 3 Sonata Software Limited

    Contents

    Abstract .................................................................................................................................................... 2

    About the Author...................................................................................................................................... 2

    Risk-Based Testing .................................................................................................................................... 4

    Generic Approach for Risk-Based Testing ................................................................................................. 5

    Statistical Models ..................................................................................................................................... 7Illustration ................................................................................................................................................ 9

    Workflow for Risk-Based approach: ......................................................................................................... 10

    Results ...................................................................................................................................................... 11

    Inferences of the concept ......................................................................................................................... 13

    Open Source Test Management ................................................................................................................ 13

    Summary .................................................................................................................................................. 14

    To read more about our views on technology, do visitwww.sonatablogs.com

  • 8/7/2019 White_Paper_RBT

    5/16

    Technical White Paper www.sonata-software.com

    White_Paper_Rbt 4 Sonata Software Limited

    Risk Based Testing

    In todays scenario, the quality of software is becoming a matter of concern. With this issue creating

    conflicting challenges, the industry is testing and trying different measures to tackle it. Innovative

    techniques, tools, technologies and ideas are being implemented to ensure availability of standard

    software. One of the popular measures adapted is Risk-Based Testing a technique through which a

    certain amount of testing can be done without covering an entire gamut of available test cases.

    According to Industry Experts, 80% of applications are either not tested or are manually tested before

    being delivered to production. This leaves the quality of such software open for speculation and hence,

    several software projects cost high due to the risks related to it.

    Although a lot of mandatory regression and end-to-end testing is being done, the earlier the defect is

    detected; the lower is the cost of solving the issues. To address these issues, Sonata has developed a

    statistical model which would provide us a required methodology for Risk-Based Testing.

    Sonata with its unparallel experience of product quality assurance services has understood that Risk-

    based testing is vital in today's competitive market. With this Risk-based approach, a reduction in cost

    per quality with a faster time to market is achieved.

    Diagram 1

  • 8/7/2019 White_Paper_RBT

    6/16

    Technical White Paper www.sonata-software.com

    Risk-Based Testing 5 Sonata Software Limited

    Risk-Based Testing

    Risk-Based Testing is a methodology which after identification of risks and their possible impact on

    system allows you to prioritize and plan your test strategy in accordance to the risk rating and mitigationplans.

    These provide us with faster time-to-market that gives us more time to fix the defects. The defects are

    not detected at the end of the release; in fact the defects can be detected in the early stages of

    Application Development itself. This is a scientific and data-based approach which results in cost

    optimization and enhancement of quality. It can identify and execute high risk data hence providing

    more time for defect fixes.

    Generic Approach for Risk-Based Testing

    Going ahead with our Risk-Based approach, a Risk Analysis is performed before starting the testing

    activities. The prime objective is to take control over the problems before problems take over the

    situation.

    The following figure shows the activities involved in Risk Analysis when a project is performed. The

    diagram below discusses this in detail:

  • 8/7/2019 White_Paper_RBT

    7/16

    Technical White Paper www.sonata-software.com

    Risk-Based Testing 6 Sonata Software Limited

    Diagram 2

    Diagram2: Risk analysis activity model - This model is taken from Karolaks book Software Engineering Risk Management,

    1996 [6] with some additions made (the oval boxes) to show how this activity model fits in with the test process.

    The first step is the Test Planning. In this phase, the risks need to be identified and a Risk Strategy should

    be created. A risk can be of many types. One of the key important types could be the complexity in the

    available applications, the type of resources and available tools. A clear Risk Strategy needs to be

    defined before getting into the Test Planning activities.

    Subsequently, the Risk Mitigation plan must be prepared. This plan clearly states for a particular type of

    risk what the risk mitigation is. For example, if it is going to be a very complex application the risk

    mitigation plan would be dissecting the application into several components as smaller modules, and

    fill up each of those components with more capable resources.

    Once the Risk Mitigation plan is completed, other important areas like the Risk Reporting can be

    focused on. Risk Reporting is very important because it provides complete transparency across the

    entire stakeholders of the project to gauge and act on the risk area. With all of the testing and

    inspection techniques and capturing all test metrics, the risks that get reported are identified. At this

    stage, one can predict the risks.

  • 8/7/2019 White_Paper_RBT

    8/16

    Technical White Paper www.sonata-software.com

    Risk-Based Testing 7 Sonata Software Limited

    In the Risk Prediction stage there is an entire set of data from which the risk-prone area is identified. In

    this model risk prediction feedback is again fed into the risk identification area and it is a cyclic process.

    By undergoing several iterations of this cyclic process the Risk Strategy as well as Risk Prediction model

    can be refined. The areas with certain defects or minimum defects or no defects can easily be predicted.

    This is the core idea of Risk Based Technology.

    Statistical Models

    In the statistical model the importance lies on the characterization of numerical data. Also, it is very

    important to estimate the probability in terms of the behavior of system.

    The entire testing activity is nothing but probability. It is the probability of finding out a particular defect

    or a particular section failing on a particular area or on a particular type of environment. These all may

    be useful in deciding the type of testing required and to ascertain the focus in areas of testing. With this

    focus, extrapolation or interpolation of the existing data can be conducted and the best fit for that can

    be identified. The best fit will provide the critical path for the defects. It will clearly provide areas that

    require testing in that particular application or system.

    There is another model called Spectral Analysis of data or model generated output which is an industry

    standard. Here the focus is on the algorithms that have been used for this Risk-Based Testing. The

    algorithm is extended to suit the current situations where the best fit for various factors is exercised.

    This helps in calculating the risk as well as the probability of failure.

    The statistical model is based on the probability of defects and a consequence of defect. These two are

    very critical in defining the Risk Exposure of the system. One of the important parameters is quality of

    the code. It may be suffering from poor designing or it may have been coded by an inexperienced

    programmer, it may be to a complex functionality. The probability of defects is defined as P(f),

    consequence of defect pertaining to the customer C(c) and consequence of defect to the vendor as C(v).

  • 8/7/2019 White_Paper_RBT

    9/16

    Technical White Paper www.sonata-software.com

    Risk-Based Testing 8 Sonata Software Limited

    Diagram 3

    Consequences of defect for a customer (which is a cost to the customer) may be:

    o The probability of a legal threat

    o Losing a market place

    o Not fulfilling regulations or FDI regulations

    The consequence of defect towards the vendor gives a negative credibility to the vendor or it would

    increase the maintenance cost because of the functions with faults.

    The combination of these two factors leads to a formula: Risk exposure [Re(f)] is calculated as a product

    of the probability of failure and the consequence effect.

    Diagram 4

    The probability of failure again characterized. It is a combination of multiple probabilities of failures.

    Normally, the consequences value weighted between 1-3, and it would count the production fault loss

    of revenue impact and incurs cost change impact also. The probability of failure is always weighed

    between 0-1.

  • 8/7/2019 White_Paper_RBT

    10/16

    Technical White Paper www.sonata-software.com

    Risk-Based Testing 9 Sonata Software Limited

    The weighted average of the probability of failure is dependent upon following factors:

    Changed functionality

    New functionality

    Design quality

    Size of the project

    Complexity

    Programmers experience

    Illustration

    The following calculation of Risk-Based matrix is with a live example from one of the Sonata's projects.

    For the sample calculation 18 factors of probability were taken into account. The customer is Europes

    largest holiday company, serves more than 23 million guests every year. Operates own resorts, hotels,

    airlines, travel agencies and cruise ships.

    The front-end was a Web Selling platform and the back-end is a mainframe system which supports the

    entire set of operations. The backend system is capable of running 600-1000 batch programs on a daily

    basis and transacts 5-6 million records every day. The entire system has around 10 interfacing systems

    (e.g. Amadeus, Alamo etc). They have multiple staging areas where they present their data and do a

    focused analysis for the next quarter or the next season coming through.

    The Risk-Based Testing was conducted for this particular engagement because there were nearly 8000-

    10,000 test cases for the entire set of Enterprise Applications. While doing so, around 2 to 3 releases are

    gathered on a monthly basis at the Enterprise Level. If a complete set of testing or end-to-end testing is

    required, it is important to cover all sets of test cases across the enterprise chain. In such scenarios, over

    2000 to 3000 test cases are run per release. This consumes a lot of effort and hence the cost, as well as a

    delay to market.

    In order to handle this situation several testing techniques are adopted, test automation in terms of tool

    level as well as in terms of data level which would also test the testing data integrity. In terms of

    approach a Risk Based Approach is followed because only a certain amount of test cases or certain

    types of test cases catch errors, rest of the test cases were defect-free. Based on our observation it was

    found that out of 2000-3000 test cases only around 200-300 test cases were capturing defects. This is

  • 8/7/2019 White_Paper_RBT

    11/16

    Technical White Paper www.sonata-software.com

    Risk-Based Testing 10 Sonata Software Limited

    because only these test cases are clearly attached to the risk prone areas. Hence, the statistical model

    was adopted to capture most of the defects and implement Risk-Based approach.

    Various kinds of testing methods have been implemented in this regard; starting with System Testing

    which involves testing of the Web selling platform, testing of their core system, testing of their business

    intelligence areas, Integration Testing, Regression Testing, Performance Testing to Security Testing etc. A

    specialized testing on Data Integrity on volume testing was also conducted. In compiling all these

    methods of testing several areas with defects were identified.

    The inputs required for fitting the statistical model are:

    o The number of defects

    o Types of defect: Database defect, Staging Area defect, Web area defect

    o

    Classification of defects: Defects originated of database, originated from the application serveror from the functionality

    o Effort required for defect identification

    o Effort required for fixing defects

    o Weightage for the probability functions in terms of failure and the consequence

    All inputs in the algorithms and routines for iterative runs were rigorously followed. This has resulted in

    the probability of failure as well as the risk exposure co-efficient.

    Workflow for Risk-Based approach:

    As the first step, all defects are classified. Once the defects have been classified the probabilities of

    various factors which affect the quality of the release are obtained, post which the risk exposure co-

    efficient is derived. Once the Risk Exposure co-efficient is identified, the co-efficients are fed into the

    iterative algorithms for various values of probability of failures. From this the type and number of test

    cases that will be utilized for Risk Based Testing are obtained.

    Example, in a live environment, if there are 2000-3000 test cases and it is known from an existing

    analysis that the 23rd or 33rd test case are going to yield defects in a particular area which comes under

    the sampling techniques, and there are other defects from a different area, then it is convenient to

    sample them on a common algorithm and feed it into algorithm. This enables the identification of the

  • 8/7/2019 White_Paper_RBT

    12/16

    Technical White Paper www.sonata-software.com

    Risk-Based Testing 11 Sonata Software Limited

    Risk exposure coefficient. Wherever the Risk exposure co-efficient is high, those are the areas that need

    100% coverage.

    In this specific example the probable failures in terms of change requests, test interfaces, inexperienced

    developer, field validations, business rules validations, positive and negative scenarios, third party

    interfaces, system integration testing, backend verifications, UI elements testing, content verifications,

    content validations, error messages verifications & validations, cross browser testing, platform

    compatibility testing, functional end-to-end flow testing have been taken into consideration. Certain

    weights have been assigned to these areas, to calculate the Risk exposure for various iterations from a

    value of 0.5 to 2.0 and to achieve a constant consequence as 2 to arrive at the probability of failure.

    Results:

    There are 3 different depictions:

    Iterations:

    Re(f) {0.5, 1.1, 1.2..2.0}

    having C(c&v) = 2

    Note: Open Source TM (Algorithms and IP) was used for this study

    Defects in Releases

    Sample: 1200 TC - Continuous

    Graph 1Defects in Releases

    Sample: 1200 TC - Random

    Graph 2

  • 8/7/2019 White_Paper_RBT

    13/16

    Technical White Paper www.sonata-software.com

    Risk-Based Testing 12 Sonata Software Limited

    Graph 1: It is a sample of 1200 test cases and it is continuous. A particular release has undergone 10-12

    iterations and the test cases have been run 1200 continuously. The test cases have been run by an

    automated test suite which has been developed on an Open Source Framework. The defects in Graph 1

    found in the different iterations and how the applications are stabilized over a period of iteration. In this

    case all the 1200 test cases on all releases in an automated way.

    Graph 2: Test cases were selected at random, without any logic or reason. This particular method also

    provides the defects but the amount of the defect captured is lesser when compared to the amount of

    defect captured while the entire set of test cases is run.

    Graph 3: The statistical-based algorithm is run and 400 out of the 1200 test cases are sampled. Only a

    minimum number of test cases are run but they capture the maximum number of defects. There is no

    variation in results as compared to the number of defects identified when all 1200 test cases are run.

    This concept saves a lot of effort and cost that leads to an impressive turnaround time.

    Defects in Releases

    Sample: 1200 TC - RBT

    Graph 3

  • 8/7/2019 White_Paper_RBT

    14/16

    Technical White Paper www.sonata-software.com

    Risk-Based Testing 13 Sonata Software Limited

    Inferences of the concept

    As a result, a 50 60% reduction on testing effort (400 test cases out of 2000 test cases) is achieved,

    generating data for multiple set of defects scenario. These defect scenarios or data will be applicable in

    subsequent releases. In a similar project or a similar type of release the respective test cases can be

    pulled straightaway. There is no impact on the critical factors that is on the database side or in the

    performance side or in the quality side by doing this.

    This has resulted in the 40-50% of the testing cost reduction. In the enterprise environment when

    multiple projects in multiple streams are run, each product needs to be tested on a particular day or a

    particular time segment. To do this testing continuously in all these areas by using risk-based approach agreater bandwidth is required to run these tests in spite of running less amount of test cases and achieve

    more amount of coverage.

    The other important advantage is Defect Predictability. Number of defects can be predicted for a certain

    size of application. This helps us in estimating the time for defect fixes. In a project lifecycle analysis,

    requirements, developments, testing and release need to be planned. Often, the element that is missed

    out is time and effort required for the defect fixes. If a decent estimate of defect fixes can be identified,

    then it is easier to estimate the time required to complete it.

    Since the developers or the programmers required for this program are selected right at the beginning,

    there can be optimized use of relevant expertise and hence the risks can be handled efficiently.

    Open Source Test Management

    In this activity of statistical model or the iterative algorithm and selecting the relevant test cases and

    then running those test cases for execution, you require a proper test management system. Either it can

    be a quality center or a QA director or any other tool which is capable of doing that. Sonata has

    developed an Open Source Test Management System which is integrated with defect and test

    management areas. It houses the entire Risk-Based Testing algorithm and the data for various values of

    probabilities of failure in different areas in terms of classification. It has become easy to do the

    automated test cases predictability and organize the test cases according to the functionality and defect

  • 8/7/2019 White_Paper_RBT

    15/16

    Technical White Paper www.sonata-software.com

    Risk-Based Testing 14 Sonata Software Limited

    areas. Customized simulation for various resources can be obtained. This reduces around 70% of the

    regression test cost and 50% improvement in controlled releases.

    Summary

    The statistical approach for Risk Based Testing is a proven model. It is capable of simulating error

    injection, analyzed impacts associated with failures. More importantly, it is simple and cost effective. As

    an added advantage an Open Source System supporting it is also available. The algorithms are scalable

    and iterative and these algorithms can be used or extended for any type of testing (Web testing, Data

    Testing, testing of ERP systems). Customized reports in terms of the available number of risk prone test

    cases are generated which definitely need to run in a particular release. All the data is available in areport format. The Open Source Test Management System houses all of these activities and functions

    together and being provided to the customer as a package.

  • 8/7/2019 White_Paper_RBT

    16/16

    Technical White Paper www.sonata-software.com

    Risk-Based Testing 15 Sonata Software Limited

    CORPORATE OFFICE

    APS Trust Building

    Bull Temple Road, N. R. Colony

    Bangalore 560 019, India

    Tel: 91-80-3097 1999, Fax: 91-80-2661 0972

    Email:[email protected]

    WORLDWIDE OFFICES

    Dubai

    Office # 507, Thurraya Tower No.1

    P O Box 502818, Dubai Internet City

    Dubai, United Arab Emirates

    Tel: 971-4-375-4355, Fax: 971-4-424-0132

    Email:[email protected]

    Germany

    TUI InfoTec GmbH

    Karl-Wiechert-Allee 4

    30625 Hannover, Germany

    Tel: 49-511-567 5296

    Email: [email protected]

    India

    6, Richmond Road

    Bangalore - 560 025, India

    Tel: 91-80-3097 3299, Fax: 91-80-2248 4045

    Email:[email protected]

    193, R.V. Road,

    Basavanagudi,

    Bangalore - 560 004, India

    Tel: 91-80-3097 2999, Fax: 91-80-2656 7487

    Email:[email protected]

    Sonata Towers, Global Village,

    Pattenegere & Mylasandra,

    RVCE Post, Mysore Road,

    Bangalore - 560 059, India

    Tel: +91-80-3097 1499

    Email:[email protected]

    1-10-176, Begumpet Main Road

    Opp. Hyderabad Public School

    Hyderabad - 500 016, India

    Tel: 91-40-3981 3899, Fax: 91-40-2776 4831

    Email:[email protected]

    Singapore

    1, North Bridge Road, #19-04/05

    High Street Center

    Singapore 179094, Singapore

    Tel: 65-633-724-72, Fax: 65-633-740-70

    Email:[email protected]

    UK

    5, Churchill Court

    58, Station Road, North Harrow

    Middlesex HA2 7SA, UK

    Tel: 44-20-8863 8833, Fax: 44-20-8863 5533

    Email:[email protected]

    USA

    39300 Civic Center Drive,

    Suite 270, Fremont, CA 94538, USA

    Tel: 510-791-7220, Fax: 510-791-7270

    Email:[email protected]

    2018 156th Ave NE, Suite 100,

    Building F, Bellevue, WA 98007, USA

    Tel: 425-372-2167, Fax: 425 484 7799

    Email:[email protected]

    1901 North Roselle Road, Suite 800,

    Schaumburg, IL 60195, USA

    Tel: 847-517-6310, Fax: 847-517-6313

    Email:[email protected]

    11330 Lakefield Drive, Bldg #2, Suite 200

    Duluth, GA 30097, USA

    Tel: 770-814-4213, Fax: 678-623-0236

    Email:[email protected]

    275 Grove Street, Suite 2-400

    Newton, MA 02466, USA

    Tel: 617-663-4866, Fax: 617-663-6127

    Email:[email protected]

    212, Carnegie Center, Suite 206

    Princeton, NJ 08540, USA

    Tel: 609-919-6325, Fax: 617-663-6127

    Email:[email protected]

    If you have any experiences related to Risk Based Testing that you would like to share with us, please

    write in to us on [email protected]

    mailto:[email protected]:[email protected]:[email protected]:[email protected]:%E0%B3%[email protected]:%E0%B3%[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:%E0%B3%[email protected]:[email protected]:[email protected]