the best pairwise testing tool / best orthogonal array tool just got better
DESCRIPTION
Powerful New Feature Enhancements now allow Pairwise testing Orthogonal array testing More thorough n-way testing Requirements traceability Seeding of high priority scenarios Automatic generation of detailed test scripts with tester instructions Automatic inclusion of rules-based Expected Results Exporting into Quality Center Exporting into mind maps And a lot moreTRANSCRIPT
![Page 1: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/1.jpg)
1
Powerful New Hexawise Features
![Page 2: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/2.jpg)
Agenda
2
• Intro and “Quickie Demo”
- What is Hexawise? What doesn’t it do?
- How does pairwise testing and related methods work?
- Advantages and Limitations of the “Old” Hexawise
• Hexawise “2.0” - The Most Powerful New Features We’ve Ever Released:
- Requirements Traceability
- More Powerful Test Design Capabilities
- Faster Test Script Documentation (including Expected Results)
- Better Auto-Scripting
• Demo of New Features
• Wrap Up with Questions and Answers
- Suggestions for Good Sources of Information
![Page 3: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/3.jpg)
What is Hexawise?
3
Define Requirements
Export Test Scripts into Test
Mgmt. Tool
It’s a test design tool.
Testers use it to do these two things better & faster.
DocumentTest Scripts
Select Test Conditions
ExecuteTest Scripts
Find Bugsand Verify Behavior
But that’s definitely not all Hexawise does….
![Page 4: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/4.jpg)
What is Hexawise? (Cont’d)
4
Export Test Scripts into Test
Mgmt. Tool
… Hexawise also improves test EXECUTION efficiency & effectiveness.
The tests you generate with Hexawise are more powerful than the tests you would select by hand. Specifically, tests generated using Hexawise are optimized to:
• Be as varied as possible,• Minimize wasteful repetition, and • Maximize test coverage in a manageable number of tests.
ExecuteTest Scripts
Find Bugsand Verify Behavior
![Page 5: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/5.jpg)
What DOESN’T Hexawise Do?
5
Hexawise is not a test management tool (like Quality Center).
It’s also not a tool that automates test execution (like QTP).
![Page 6: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/6.jpg)
When and Where is Hexawise used?
6
Export Test Scripts into Test
Mgmt. Tool
ExecuteTest Scripts
After testers generate tests in Hexawise, they usually export them to a tool like Quality Center.
Hexawise generates tests for teams that do both: - Manual testing, and - Automated testing.
![Page 7: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/7.jpg)
Poll Results: Most Popular Hexawise Benefit?
7
How? Test generation algorithms maximize testing coverage.
DocumentTest Scripts
Select Test Conditions
ExecuteTest Scripts
Find Bugsand Verify Behavior
How? If you want to, you can execute fewer, more powerful tests.
How? Hexawise automates many test design and test
documentation steps.
1. You document tests faster…
2. You complete test execution faster…
3. You consistently find more bugs.
4. Your team makes more informed, data-driven decisions.
How? From much more precise coverage analysis reporting.
![Page 8: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/8.jpg)
Where Do Testers Use Hexawise?
8
Everywhere.
![Page 9: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/9.jpg)
Is Hexawise Applicable to My System?
9
Yes.
“When testing anything: "No matter what kind of interface it has, you can always identify interesting things to vary."
- Elisabeth Hendrickson – in her brand new book “Explore It!”
![Page 10: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/10.jpg)
Agenda
10
• Same Easy to Use Features as Always, to Increase Your…
- Test selection speed
- Test documentation speed
- Testing thoroughness• “Quickie Demo”
- Advantages and Limitations of the “Old” Hexawise• Plus The Most Powerful New Features We’ve Ever Released:
- Requirements Traceability
- More powerful Test Design Capabilities
- Faster Documentation with Expected Results
- Better Auto-Scripting• Demo of New Features
![Page 11: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/11.jpg)
Charlie’s Chocolates – Release 7
11
The “interesting things to vary”…
![Page 12: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/12.jpg)
12
Select Test Conditions
Simply (A) Enter test conditions, and (B) click here.
As before, to do this
![Page 13: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/13.jpg)
How does Hexawise prioritize tests?
Short version:
1) Pairwise: In this example – using the default “pairwise” coverage strength - Hexawise will select 16 tests from 576 total possible ones. Those 16 tests include at least one test of every possible pair of Values.
2) Rationale: The vast majority of your defects that sneak by undetected into production will be triggered if you tested for every possible pair of Values.
3) Manually-Selected Tests Have Many Holes: If you started to test this way tomorrow, even with fewer tests, you would find many more defects than you currently do because you would be closing thousands of gaps that exist today.
4) First Draft vs. Final: These 16 are a “first draft” set of powerful tests. Not a final set. You can and should modify them based on your knowledge.
5) Turn the Coverage Dial Up or Down: You can easily generate however many tests you want depending upon your time vs. thoroughness preferences: a dozen?, a hundred?, all 576?
For more information, data, and real-world case studies, see: pairwisetesting.com
13
![Page 14: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/14.jpg)
14
… testers would transform powerfully optimized sets of test conditions like these…
DocumentTest Scripts
Before, to do this
![Page 15: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/15.jpg)
15
…into “test scripts” like these below using the Auto-Script feature. (Note that these have both: (A) detailed instructions and (B) test conditions).
![Page 16: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/16.jpg)
Advantages & Limitations: “Old” HexawiseTest Script Documentation:
While we were able to
Quickly select and document new test scripts…
And re-write the tester instructions “instantly” for each test…
Those test scripts…
Did not include Expected Results.
16
![Page 17: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/17.jpg)
Advantages & Limitations: “Old” HexawiseRequirements / Ability to Include Specific Complex Combinations:
While we were able to
Quickly generate new sets of highly-varied test scripts…
And those tests had maximum coverage possible with minimum repetition…
We did not have the ability to…
Include specific combinations of Values to test for our
complex requirements.
17
![Page 18: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/18.jpg)
So What’s New in “Hexawise 2.0”?
18
![Page 19: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/19.jpg)
Charlie’s Chocolates – Release 8Let’s see how “Hexawise 2.0” handles changes for the next release:
New Features to Test in Release 8 of Charlie’s Sales Platform:
1. Charlie now accepts MasterCard
2. Charlie now offers 2-day shipping
Additional “Trouble Area” to Test More Thoroughly in Release 8:
3. Single item orders are working well; when multiple items of an item are
ordered, odd things have been happening in Release 7.
19
![Page 20: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/20.jpg)
20
We need to add some new inputs (as usual)…
![Page 21: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/21.jpg)
… which will lead to an entirely new set of “scrambled” tests which need new ERs.
21
… tons of new Expected Results!
![Page 22: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/22.jpg)
Good news: we can now generate Expected Results automatically!
22
![Page 23: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/23.jpg)
Which means creating test scripts like these is now easy and fast:
23
Once you generate your simple rules, hundreds (or thousands!) of these Expected Results will be generated with the push of a button… without a single mistake.
![Page 24: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/24.jpg)
So far so good, but how can we test a convoluted warning message like this?!
24
http://37signals.com/svn/posts/3239-a-great-customer-experience-nutscom-goes-the-extra-mile
This “warning” feature can only be tested by specifying many specific Values to appear in a test.
This example (a result of both great customer service and good testing) was “seen in the wild” by one of my heroes:
![Page 25: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/25.jpg)
Charlie’s Chocolates – Release 8And how does “Hexawise 2.0” handle these additional testing goals?
Two Requirements to Test (from new Business Rules):
1. Fizzy Lifting Drinks ordered on November 11 should get an 11% discount.
2. “The complex conditional melted chocolate notification”
• If customer selects 2-day delivery,
• … and the type of food being shipped is made of chocolate,
• … and it would actually take 3-days to deliver (b/c of a Sunday or Holiday),
• … and it is predicted to be hot enough to melt chocolate outside,
• …. Then notify the customer there is a risk the chocolate might melt.
25
![Page 26: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/26.jpg)
Here’s how to think about entering Requirements in Hexawise:
26
Ask “what is the Expected Result of the requirement?”
Ask “What is the smallest number of test inputs you need to specify in order to see if the requirement is true or not?”
Enter those test inputs (and only those) into the Requirements screen.
First –
Second –
Third –
![Page 27: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/27.jpg)
Requirements
27
“Fizzy Lifting Drinks ordered on November 11 should get an 11% discount.”
1. Expected Result =?
“should get an 11% discount.”
2. Smallest Number of Values to trigger the discount = ?
2:
Type of Candy must = Fizzy Lifting Drinks
Date Ordered must = November 11th.
3. Now just enter those two Values (and only those two
values!) into the Requirements screen and enter the ER!
![Page 28: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/28.jpg)
Add the Values in Screen:
28
![Page 29: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/29.jpg)
Requirements Feature (Cont.)
29
How many specific Values do we need to trigger it?
4. So simply select these four values from the drop down menus
on the Requirements screen and add your Expected Result:1. Customer selects 2-day delivery, and
2. The type of food being shipped is made of chocolate, and
3. It would actually take 3-days to deliver (b/c of a Sunday or
Holiday), and
4. A weather prediction system predicts it will be hot.
What is the Expected Result we want to trigger?
“Then notify the customer there is a risk the chocolate might melt.”
To add a test for the second “complex melting chocolate notification”
requirement, we follow the same process:
![Page 30: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/30.jpg)
“Traditional” Recommended Sources
Explanations of how Pairwise Testing (and similar, more sophisticated test design methods) consistently pack more testing coverage into a manageable set of tests than hand-selected tests
• pairwisetesting.com
Questions about features (e.g., “How do I…?”)• help.hexawise.com
Training / understanding test design concepts• training.hexawise.com
An introductory, fact-based article for skeptical managers• A scholarly, evidence-based
article and explanation of benefits from a 10 project study that I co-wrote with 3 PhD’s
30
![Page 31: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/31.jpg)
Fun Recommended Sources
• A fun presentation about pairwise software testing - powerful test design
• Nutty video announcement of our new “Hexawise 2.0” features
31
http://www.youtube.com/watch?v=3F1eQag6XUg
hexawise.com/?p=531
![Page 32: The Best Pairwise Testing Tool / Best Orthogonal Array Tool Just Got Better](https://reader033.vdocument.in/reader033/viewer/2022061206/548261d2b07959240c8b47af/html5/thumbnails/32.jpg)
Sign up for a free account now:
32
hexawise.com/free
Try Hexawise out yourself.