combinatorial testing - mstb gack... · combinatorial testing mba, six sigma black belt, asq...

1© 2011 Process-Fusion.net

Kuala Lumpur, Malaysia

Gary A. Gack

Owner, Process-Fusion.net

[email protected]

Combinatorial Testing

MBA, Six Sigma Black Belt, ASQ Certified Software Quality Engineer

… a ‘hands-on’ workshop

Agenda

© 2011 Process-Fusion.net 2

What is “Combinatorial Testing”?• an introduction to terms and concepts

Interactive “Case Study” Demonstration• the “Hexawise” tool

Hands-On Workshop• Review and discuss test problems to be used during the workshop• “Let’s get to work using the tool”

Review and de-brief plans created during the session

Overview of other similar tools

What is “Combinatorial Testing”?


The objective: Maximize “learning”, minimize “investment”– e.g., test various drug formulations; find optimal wing design

The challenge: potentially millions of “permutations”– e.g., 5 ingredients, each ranging from 1% to 100% concentration

– In 1% increments, potentially 100*100*100*100*100 = a big # (100**5)

– This option is called a “full factorial design” (rarely feasible)

The solution: focus on “combinations” of factors, restrict “levels”– e.g., 3 levels for each factor (hi/med/lo)

– Check combinations of factors/levels

– This option is called a “fractional factorial design”


2-way (or “n-way”)

Design of “Experiments”: The Concept

Combinations of factors reflect business rules

… if a and b then (whatever)

Studies indicate 85% of defects (on average) arise from 2-way interactions

… progressively fewer are found as n-way increases

5

Source: “Combinatorial Testing” IEEE Computer, August, 2009. Dr. Rick Kuhn, Dr. Raghu Kacker, Dr. Jeff Lei, and Justin Hunter.

© 2010 Process-Fusion.net

Hexawise provides a systematic way to identify the minimum number of tests necessary to achieve n-way coverage

Why “Combinatorial” (n-way) Interactions Matter


This was a because it always happened whenever a user’s mouse was over a plan name. The bug would happen (regardless of browser type, type of user, which of the three main screens of the application the user was on, what the user was doing, etc.)

A hand icon should appear here, not a pointer.

single-mode fault

Example of a single-mode fault:

Why “Combinatorial” (n-way) Interactions Matter


Example of a dual-mode fault (AKA “2-way fault”):– Users could share plans with one another– Users could rename plans– However, users could not rename plans that were shared with them

It is a dual-mode fault because it requires two different values to be present in order to occur. The only kind of test that uncovers these kinds of defects is one that combines both aspects in the same test case. In this case, a single test would need both:

- Plan type = a plan that has been shared with the user

- Action = rename the plan


Example of a 3-way or 4-way fault?

They are rare.


Example of a 3-way fault

Skitch, a screen capture tool, just released a fix for this rare triple-mode bug:

To find this bug... all three of the following needed to occur together

1. Type of image = .jpg, and2. O/S = Mac Snow Leopard, and3. Method of inserting image = “dragging”

Even if you had isolated the 3 parameters that were contributing to the cause, it might have taken 16 attempts to recreate the defect / confirm its cause:

Scenario Type of Image Mac O/S Method of Inserting an image Result

1 .jpg Snow Leopard Cut and Paste “Works on my machine”

2-8 .jpg 7 earlier versions Drag and drop “Works on my machine”

9-15 7 other formats Snow Leopard Drag and drop “Works on my machine”

16 .jpg Snow Leopard Drag and drop Finally found the cause

“If I try this test I may find a defect”

A Simple Example – 4-factor configuration testingOperating system (XP, W7, Linux)CPU (1.5, 2.0, 2.5 Mhz)Protocol (IPv4, IPv6)Memory (1mb, 2 mb, 4 mb)

full coverage requires 54 tests2-way requires 11 tests3-way requires 27 tests


Testing is Essentially an Experiment

Combinatorial Methods “Front-Load” Tests… when should we stop testing? How much is enough?


27 factors

2-6 levels each

Potentially

2-way = 46 tests

3-way = 299 tests


26,418,075,402,240 tests!

A More Complex Example – Expedia reservation

Find More Bugs During Testing!… on average 2.4x more

bugs per tester hour… 13% more in total

Reduce Testing Cost… average 30-40%

reduction in test planning & execution effort


The Business Case

14

This single web page could be tested with

TM

There are way too many options to test every one. What should we test?

1515

?

Testing each feature to “see if it works” is not enough.Does it work in combination with every other feature?

1616

?

Questions

1717

?

Testing each feature to “see if it works” is not enough.Does it work in combination with every other feature?

19

Automated test condition generation

100% coverage of important combinations, achieved efficiently

Unchanged (still manual / still dependent upon thoughtful SME’s)

What Changes? What Doesn’t?

2 Create Tests

1 Define Inputs

3 Analyze Coverage

Impact of Combinatorial Method

Interactive “Case Study” Demonstration

The Hexawise Tool


• This two part example provides an illustration of how Hexawise might be used to test portions of Amazon.com. Hexawise can be used at different levels of abstraction

– Part 1 of this example shows how Hexawise might be used in the early stages of testing to “explore” the application.

– Part 2 examines a “deeper dive” into a limited area of the application.


Case Study: Amazon.com

• “All testing efforts require exploratory testing at some time or another, whether test procedures are based on the most detailed requirements or no requirements are specified.

– Exploratory tests cannot be pre-planned, as they are conducted on a case-by-case basis in reaction to encountered conditions.

–Elfriede

Dustin, 2003

• The example we consider here illustrates how a portion of an exploratory test might be thought through and documented before tests cases are executed. I agree with Dustin that exploratory tests cannot always be fully defined in advance, but as we illustrate here there is often a middle ground.


However, it is a good practice to document the exploratory tests before or after execution, adding them to the suite of regression tests…” (emphasis added) –

Effective Software Testing,

Part 1: Exploratory Testing

We may begin with several questions or hypotheses in mind, e.g.:

• We wonder if the system behaves differently when we select items stocked by Amazon, such as new books, compared to items not stocked, such as used books and many electronics.

• We want to be on the lookout for areas that are potentially high risk – areas where defects could have serious consequences.

• We want to see if there are differences related to the more common configuration options.


1) Identify the objectives and scope of your testing

• We do not plan to test exception conditions – i.e., situations that should generate error messages or cause the application to terminate.

• We do not intend to try every conceivable path – we want to focus on the most common. In this illustration we will limit ourselves to – searching for books or CDs, – adding items to our cart, – checking out, and – canceling our order. – For now we will not explore other functions of the system


2) Identify what will be excluded from this round of tests

• Often we begin by enumerating any relevant configuration options – e.g., PC and Mac; Firefox, IE 6-7-8, and Chrome.

• Next we think about the major functions of the system– e.g., in Amazon we can find a book or a CD, we can add or remove

items to our cart, and we can checkout or cancel.– Notice we can leave a lot of discretion to the tester. We will ask

them to select both new and used books to confirm or reject our hypothesis related to stocked or non-stocked items, but we are not specifying any item specifically.

– We are also not indicating anything about payment methods or other options that arise during checkout.


3) Select test inputs and enter them into Hexawise

4) Determine testspotential


2 x 5 x 2 x 2 x 2 = 80 potential tests

• Hexawise algorithms select the most powerful 10 tests from among the 80 possible tests.

• In short, Hexawise generates a set of tests, so that:– Maximum coverage is achieved in as few tests as possible– The tester’s coverage objectives are achieved (whether the tester

instructs it to create a thorough set of 2-way tests or a phenomenally thorough set of 6-way tests)

– Inefficient repetition from test case to test case is scientifically minimized

• This does not mean we are telling our testers to necessarily limit themselves to just these 10, but we are suggesting these 10 be executed first.


5) Which tests you should execute first?

“Top 10” 2-way tests


• Every Parameter Value will be tested in at least one of the tests.

• Every valid pair of Parameter Values is tested together in at least one test case. In other words, every Parameter Value is tested in combination with every other Parameter Value in at least one of the tests.

• These particular 10 tests are especially powerful because the vast majority of defects can be triggered by just one or two test inputs.

• Decreasing marginal returns – It is quite uncommon for a defect to only be triggered by a specific combination of three or more test inputs. If our objective is good coverage at minimum cost, we won’t focus our testing on those cases unless we have extra time after executing these higher priority 2-way tests. In safety critical domains, we would typically go for more thorough coverage.


6) Why those particular 10 tests?

• No differences related to the configuration parameters were identified.

– We decide for our next round of tests we will not concern ourselves with these variables.

– Most likely we will include them in a round of regression tests near the end of the test cycle, but in the interest of efficiency we will not include them in the earlier rounds of tests.

• We see that the checkout and pay area has significant financial consequences and is therefore a high risk area – we want to test that thoroughly.

• We did not see any differences related to stocked or non-stocked items, so we do not intend to give that any special attention in the next round of tests.


Assume that the results of this exploration were the following:

• The example we consider here continues testing of Amazon.com with a specific focus on one subset of the available functionality.

– We often find it is useful to “segment” an application into a series of smaller focus areas, both because that reduces the scope of the test plan and it allows different testers to focus on different areas of the application in parallel.

– This is a particularly useful approach when the system under test provides different sets of functionality based on the “state” the system is in at a given point in time. The Amazon checkout function, for example, functions only when the cart contains at least one item.

• Another interesting aspect of this case is that it illustrates how you might choose to navigate the eternal debate “to script or not to script”.

– Some testing gurus believe it’s a waste of time to script tests, while others believe scripting thoroughly leaves less to chance.

– My view is “all things in moderation”. You really don’t need to choose one way or the other. As this case illustrates, certain things can be scripted and others left to tester discretion within the same set of test cases – the best of both views!


Part 2: A Deeper Dive

1. Please register and log on to the Hexawise web site– Individually or in pairs (no more than 2 please)– https://app.hexawise.com/


How this section will work

Sign-up continued


Sign-up continued


Please use this email address to login to your account:

Subject: Please activate your new Hexawise accountWelcome to Hexawise!Thank you for registering for access to Hexawise and checking it out. We are happy to have you as a member of our community and we welcome your feedback and suggestions on how to improve Hexawise. Your account has been created. Please visit this Web page to activate it (this validates your email address):

https://app.hexawise.com/activate/e11f61d389a322ccc8584c62880f88ef02fca67cYou are on your way to creating more coverage with fewer tests.

2. Create a new plan


How this section will work

• Our test objective in this example is to thoroughly test all of the options available to us when we decide we are ready to checkout and complete the payment process.


1) Identify the objectives and scope of your testing

• One or more items have been selected and added to the cart when this test begins. Testing of search and select functionality is not within the scope of this test.

• We will not select any of the available help options during this test.

• We will not test this functionality with any of the configuration options (Mac/PC, browser type, javascript, etc.)

• The final “Place Order” screen allows us to change shipping address, method of payment, method of shipment, etc. We will instruct the tester to try each of these options with manual (unscripted) tests. These options essentially stand alone and do not interact with the primary functionality.


2) Identify what will be excluded from this round of tests

• We will create two similar but different sets of inputs. The first set will explore the “New Customer” scenario. Executing this set of tests will, incidentally, create some of the data needed to test the “Returning Customer” scenario. As described in the following, these scenarios are in a number of respects identical, so we can use the Hexawise capability to copy a plan (New Customer), rename it, and then edit it to create a new plan (Returning Customer).


Theunfolds as follows: a) The first screen we encounter when we select Checkout asks for our email address and requires we indicate if we are a new or returning customer.


New Customer Scenario



b) As a new customer we must complete a Registration by providing our name, email address, and password. We then click Create account.

c) Which requires we enter a name, address lines 1

and 2, city, state or province, ZIP, country, phone, and a button to indicate if this address is also our billing address.

Inputs we have identified so far …


“passive”

“active”(Value Expansions)

• As mentioned earlier, we have elected a middle ground on the scripted or unscripted question – we’ve include some elements of both approaches in this example. Depending on your preferences and the specifics of your situation you might have chosen an approach that is either more or less scripted – here’s what those options might look like:


Notes

Now, enter these parameters and values


Now let’s pick up where we left off …


This screen also contains a check box to indicate gift box options:

This is an option we decide we will not script, but we will instruct the tester to try this at least once to make sure the charge for gift wrap appears on the final checkout screen.

… select a shipping address


Here we have the option to select the shipping address previously entered, we can edit that address, delete it, or enter a new shipping address. We will instruct the tester to try these options on a few test cases, but we will not script the details.

We then specify our method of payment :

… specify our method of payment


We want to script this so that each of the available payment options is tested, but we are not going to test the “Apply now” or “Sign up now” options for Gift Cards or for the Amazon.com Store Card.

Notice these payment options are mutually exclusive – we must choose one or another. To accomplish this with Hexawise we use the “Invalid Pairs” feature. More about that below.

Finally! We’re now ready to actually


This screen gives us various options – we can change payment methods, shipping address, gift options, shipping method, and we can enter promotional or gift card numbers. We will instruct our tester to try these options a few times, but we decide we will not script the specific values to be entered.

Enter the following additional parameters and values:


Now, enter the remaining parameters and values


• As mentioned above, the three available payment options are mutually exclusive – in a given test case only one method can be chosen. We want to test all of the options, but we must exclude two of the three in any single test case.

• We do that by using the “Invalid Pairs” feature. When we use this feature we must always ensure that at least one of the values for each parameter is a “valid” pairing – combinatorial logic will not allow us to exclude every possible choice even though, in effect, that is exactly what we want to do. We overcome this constraint by defining a value we call “not applicable” as the “valid” choice for each of the payment options. “Not applicable” tells the tester to ignore this input as it will not actually be allowed by the system. Invalid pairings for each method of payment are illustrated here.


Additional notes

Now, enter the invalid pairs


• We noted above that we would use the Value Expansions feature to identify states that require collection of sales tax. The definition for “taxable” looks like this:


Additional notes

• When you have entered these, click on


Next, create the value expansions

•

• Even though this is clearly an application that must have a very high level of reliability and must stand up to a large volume of transactions, it is very unlikely there will be enough time and resources available to actually perform 12 million tests. We will want to be thorough, and Hexawise can help us achieve that with many fewer tests.


Seems pretty simple, but even this limited set of parameters and values requires 12,441,000 test cases for complete coverage of every combination.

4)

• It takes Hexawise algorithms just seconds to select the most powerful 33 tests from among the 12 million+ possible tests. Hexawise generates a set of tests, so that:

• Maximum coverage is achieved in as few tests as possible

• The tester’s coverage objectives are achieved (whether the tester instructs it to create a thorough set of 2-way tests or a phenomenally thorough set of 6-way tests)

• Inefficient repetition from test case to test case is scientifically minimized


5) We will begin with 33 most powerful tests of 2-way interactions

• Given the nature of this application – high volume financial transactions – we will most likely employ a strategy of progressively higher combinatorial coverage over a series of test iterations.

• First we will execute all 33 2-way tests. This set of test cases is likely to find 60-85% of defects present.

• After we have run all of 2-way set we will begin to run the 3-way set. Assuming the 3-way reveals more defects we will then progress to 4-way coverage.


We will begin with 33 most powerful tests of 2-way interactions

• Every Parameter Value will be tested in at least one of the tests.• Every valid pair of Parameter Values is tested together in at least one

test case. In other words, every Parameter Value is tested in combination with every other Parameter Value in at least one of the tests.

• These particular 33 tests are especially powerful because the vast majority of defects can be triggered by just one or two test inputs.

• Decreasing marginal returns – It is quite uncommon for a defect to only be triggered by a specific combination of three or more test inputs. If, as is often the case for business type applications, our objective is good coverage at minimum cost, we won’t focus our testing on those cases unless we have extra time after executing these higher priority 2-way tests. In critical application domains, such as this one, we would typically go for more thorough coverage.


6) Why those particular 33 tests?

• Hexawise uses a sophisticated statistical technique to find the minimum number of test case that will guarantee a desired level of combinatorial coverage. More impressive still:

• Hexawise can help you answer “How much testing is enough?” and “When should we stop testing?”

• With Hexawise, you can make fact-based decisions about these important questions.

• The most important test cases (the ones that achieve the most coverage) are placed in priority order. Inevitably, a decreasing marginal return will occur over the course of a test plan.


7) Front-loaded test coverage

• Here, for example, we would achieve coverage of 90% of the total number of 3-way combinations that exist in our testing scope with 86 of the 173 test cases identified.

• If, for example, we had discovered 9 defects with those 86 tests it is statistically (not certain) that we will find only one more is we execute the additional 87 test cases. This is very valuable information not usually available – management can make data-based risk decisions about when to stop testing.


probable

Front-loaded test coverage

• How many tests are required to reach 80% coverage in the 2-way case?


Click on

• We now have a test plan that we feel is complete.

• Depending on our knowledge of the application and our level of confidence that we’ve covered everything important we will often review the plan with one or more subject matter experts and perhaps with management and/or our internal business customers.

• Let’s assume here that we get the green light to go ahead as we’ve planned so far.


8) Review the plan with subject matter experts and stakeholders

• We use the Hexawise “Export” feature to create an Excel spreadsheet that will contain a row for each test case, and a column for each parameter. The intersecting cell will contain specific values (in some cases indicating tester discretion). Here’s what the first 2-way test case (of the 33 created) will look like in this example (actually all in a single row, but for readability segmented here):


9) Prepare handoff to testers

Prepare handoff to testers – “Export”


We use the Hexawise “Export” feature to create an Excel spreadsheet that will contain a row for each test case, and a column for each parameter. The intersecting cell will contain specific values (in some cases indicating tester discretion). Here’s what the first 2-way test case (of the 33 created) will look like in this example (actually all in a single row, but for readability segmented here):

Now, try the Export


• Notice each row will differ in certain respects from every other row – no test case is redundant.

• In practice this is very difficult to achieve with traditional test design methods.

• Also notice that Hexawise substitutes actual values for “placeholders” when value expansions have been used – e.g., in this first case the value “Kansas” replaces the placeholder “taxable”.

• We also note certain values are left to tester discretion – i.e., when “any” is indicated. Many of these, such as “ship to address 1” have no impact on system functionality.

• Across the set of test cases generated we can be 100% confident that every “n-way” (2-way here) combination has been included.


Notes

• If we are dealing with very experienced testers this export might be sufficient. It might be used to execute tests manually, or we might use one of the Hexawise tools that support importing these tests cases into popular test execution tools.

• In many cases this leaves a lot of room for misunderstanding and resulting delays or errors during test execution.

• When we feel it is desirable to provide more guidance we will use the Hexawise “Auto-script” feature.


Notes

• The “Autoscript” feature provides a mechanism to give the tester a set of rather specific instructions that either tell the tester exactly what to do in each test case or alternately indicate discretion and provide only general guidance.

• This is actually a very powerful feature that is especially useful when you are handing off to a remote group – if they are half way around the globe this feature may help you avoid calls at inconvenient hours.

• Once you get the hang of it’s easy to use, but as it has a number of “moving parts” this explanation will cover it in small steps. We’ll focus here on what this function does, and a little later I will show you the “how to” in Hexawise.


10) If desired, prepare tester instructions

Autoscripting has three parts:

• Start – this part allows you to tell the tester anything they need to do before beginning execution of the test cases provided.

– to test the Amazon checkout function one or more items must be in the shopping cart

• Test instructions. – specific test case by test case and parameter by parameter guidance we decide to

provide to the tester. They may include expected results.

– instructions may be very explicit or they may grant discretion to the tester and provide only general guidance. In many cases, as in the Amazon checkout case, we provide a mix of these extremes.

– Hexawise is especially powerful because it provides a way to write an instruction once and automatically generate a tailored version of that instruction for each individual test case – write it once, use it many times

• Finish – this part might define any actions to be taken when all test cases have been executed. We might, for example, instruct the tester to make a copy of an end state database and restore it to the original state.


If desired, prepare tester instructions

• . Selected portions of the script for the first generated test case are shown below:


If desired, prepare tester instructions

• You may recall we mentioned earlier that the plan we have described so far covers the “new customer” scenario.

• We also mentioned that the “returning customer” scenario is largely the same, but requires fewer steps.

• The simplest way to create a test plan for the returning scenario is to make a copy of the plan we created for “new” and simply make minor changes to it.

• We’ll not go through the details of that as I am confident you could now do that without repeating the obvious.


11) Copying, Sharing, and “Cloning” Plans

“Let’s Get to Work!”


Hands-On Workshop

• What is the application?

• What is the current status?

• Do you think combinatorial testing is applicable?

• Don’t have an application?– Try creating a test for a web site you are familiar with– For example, airline reservation, on-line store


Describe your test situation

• Raise any questions as they occur to you


Begin to create your test plan


Review & DebriefPlans Created

Overview


Similar Tools

Available Tools


http://www.pairwise.org/tools.asp

Available Tools


free

Available Tools


2-way test cases generated



Thank You!

terima kasih

謝謝

combinatorial testing - mstb gack... · combinatorial testing mba, six sigma black belt, asq...

Documents