![Page 1: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/1.jpg)
1
Conquering the Single Largest Challenge Facing
Today's Testers
Justin HunterCEO of Hexawise
Tuesday, March 5, 13
![Page 2: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/2.jpg)
2
“Bad News”Many slides in this presentation might not make sense on their own.
“Good News”The YouTube video taken of this presentation is available here.
Photo credit on opening slide: it’s my own photo (taken in the fantastic Rankapur Jain temple in Rajasthan) http://www.flickr.com/photos/82153534@N05/8515803041/sizes/o/in/set-72157632883997160/
Tuesday, March 5, 13
![Page 3: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/3.jpg)
3
“There’s too much to test and not enough time to test it all.”
According to a recent survey conducted by Robert Sabourin, this is the single largest challenge facing test managers today.
Single Biggest Challenge
Tuesday, March 5, 13
![Page 4: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/4.jpg)
4
1. What Happened?
2. Avoidable?
3. Practical Implications
I. “Maps Mayhem”
Tuesday, March 5, 13
![Page 5: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/5.jpg)
Here...Most Careers
TimeTuesday, March 5, 13
![Page 6: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/6.jpg)
Here...Most Careers
TimeTuesday, March 5, 13
![Page 7: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/7.jpg)
Time
Here...Scott’s Career
Tuesday, March 5, 13
![Page 8: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/8.jpg)
Time
Here...Scott’s Career
Tuesday, March 5, 13
![Page 9: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/9.jpg)
Time
Here...Scott’s Career
Tuesday, March 5, 13
![Page 10: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/10.jpg)
Here...Even worse...
9 / 10
Tuesday, March 5, 13
![Page 11: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/11.jpg)
9
2nd to Go (He’s also Amazing)
Tuesday, March 5, 13
![Page 12: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/12.jpg)
Here...Nightmare Worsens
123,000
Tuesday, March 5, 13
![Page 13: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/13.jpg)
Here...CEO’s Apology Letter
“We are extremely sorry...”“While we’re improving Maps,
you can try alternatives... like
Bing, MapQuest and Waze, or
use Google or Nokia maps...”
Tuesday, March 5, 13
![Page 14: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/14.jpg)
12
Everyday Fails (Cont.)
Tuesday, March 5, 13
![Page 15: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/15.jpg)
13http://www.itsagadget.com/2012/09/apple-google-maps-ios-6.html
Missing Details
Tuesday, March 5, 13
![Page 16: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/16.jpg)
14
Squiggly Roads
http://www.fastcompany.com/3003446/apple-reportedly-fires-their-maps-man
Tuesday, March 5, 13
![Page 17: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/17.jpg)
15
http://news.yahoo.com/blogs/technology-blog/apple-ceo-apologizes-maps-recommends-google-instead-182143889.html
On the water
Tuesday, March 5, 13
![Page 18: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/18.jpg)
16http://machineslikeus.com/news/get-lost-apple-maps-road-nowhere
In the water
Tuesday, March 5, 13
![Page 19: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/19.jpg)
17
Missing water
Tuesday, March 5, 13
![Page 20: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/20.jpg)
18http://theamazingios6maps.tumblr.com/page/6
Water Turned into Beaches
Tuesday, March 5, 13
![Page 21: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/21.jpg)
19http://theamazingios6maps.tumblr.com/page/4
Melted Streets
Tuesday, March 5, 13
![Page 22: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/22.jpg)
20
http://www.crowdsourcing.org/images/resized//editorial_19902_780x0_proportion.jpg?1349379876
Social Media Mockery...
Tuesday, March 5, 13
![Page 23: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/23.jpg)
21
http://blogs.telegraph.co.uk/technology/micwright/100007771/apple-moronic-new-maps-this-is-turning-into-a-disaster/
Even Mocked by These Guys!
Tuesday, March 5, 13
![Page 24: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/24.jpg)
22
http://www.businessinsider.com/google-maps-apple-maps-2012-10
Impact to Sales?
Tuesday, March 5, 13
![Page 25: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/25.jpg)
23
In Fairness to those Involved
- Extreme Complexity - Unimaginably Large Scope - Highly Visible Mistakes - Google Had a Huge Head Start
http://img.photobucket.com/albums/v40/Dragonrider1227/chainsawsonfire.jpg
Tuesday, March 5, 13
![Page 26: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/26.jpg)
24
Could this have been Avoided?
Imminent DisasterSometimes it’s just better to grab a beer and watch...
Tuesday, March 5, 13
![Page 27: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/27.jpg)
25
I. More Smart Testers
This man, Harry Robinson, is a genius.
He helped lead testing for Google Maps.
IMO, he’d be a bargain to Apple at $1 million / year.
http://model-based-testing.info/2012/03/12/interview-with-harry-robinson/
Tuesday, March 5, 13
![Page 28: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/28.jpg)
26
II. Using Smart Test Design
6 browser choices
x 3 options x 2 options x 2 options x 2 options
x 4 options x 2 options x 3 options x 2 options
x 2 options = 13,824 possible tests...
...13,824 possible tests x 4 options x 4 options x 4 options
= 884,736 possible tests...
...884,736 possible tests x 5 optionsx 2 optionsx 2 optionsx 2 optionsx 2 optionsx 4 optionsx 2 optionsx 2 optionsx 2 optionsx 4 optionsx 2 optionsx 2 optionsx 2 options
72,477,573,120 possible tests
This single web page could be tested with
25
Tuesday, March 5, 13
![Page 29: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/29.jpg)
27
II. Using Pairwise Test Design
Tuesday, March 5, 13
![Page 30: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/30.jpg)
First, users input details of an application to be tested...
TM
28
Tuesday, March 5, 13
![Page 31: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/31.jpg)
Next, users create tests that will cover interactions of every valid pair of values in as few tests as possible.
(1) Browser = “Opera” tested with (2) View = “Satellite?” Covered.(1) Mode of Transport = “Walk” tested with (2) Show Photos = “Y”? Covered.(1) Avoid Toll Roads = “Y” tested with (2) Show Traffic = “Y (Live)” ? Covered.
(1) Browser = IE6 tested with (2) Distance in = KM and (3) Zoom in = “Y” ? That is a 3-way interaction. It might not be covered in these 35 tests. See next page.
29
Tuesday, March 5, 13
![Page 32: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/32.jpg)
The third screen provides objective coverage data which is useful in determining when to stop testing.
% Coverage by Number of Tests100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
2 4 7 9 11 14 16 18 21 23 25 28
Every test plan has a finite number of valid combinations of parameter values (involving, in this case, 2 parameter values). The chart below shows, at each point in the test plan, what percentage of the total possible number of relevant combinations have been covered.
In this set of test cases, as in most, there is a significant decreasing marginal return.
30
Tuesday, March 5, 13
![Page 33: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/33.jpg)
Testing each feature to “see if it works” is not enough.Does it work in combination with every other feature?
Tuesday, March 5, 13
![Page 34: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/34.jpg)
32
Every pair of test inputs get tested in at least one test!
Tuesday, March 5, 13
![Page 35: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/35.jpg)
Prioritization
How many test inputs are needed to trigger defects in production?
51%33%
11%5%
• Medical Devices: D.R. Wallace, D.R. Kuhn, Failure Models in Medical Device Software: an Analysis of 15 Years of Recall Data, International Journal of Reliability, Quality, and Safety Engineering, Vol. 8, No. 4, 2001. • Browser, Server: D.R. Kuhn, M.J. Reilly, An investigation of the Applicability of Design of Experiments to Software Testing, 27th NASA/IEEE Software Engineering Workshop, NASA Goddard SFC 4-6 December, 2002 . • NASA database: D.R. Kuhn, D.R. Wallace, A.J. Gallo, Jr., Software Fault Interactions and Implications for Software Testing, IEEE Trans. on Software Engineering, vol. 30, no. 6, June, 2004. • Network Security: K.Z. Bell, Optimizing Effectiveness and Efficiency of Software Testing: a Hybrid Approach, PhD Dissertation, North Carolina State University, 2006.
1
2 (“pairwise”)
3
4, 5, or 6
Tuesday, March 5, 13
![Page 36: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/36.jpg)
This part of the plan involved the following 6 parameters, each of which had between 2 and 4 values:
Banking / Capital Markets Case Study
34
A
09
I
C
A
V
B
10
R
E
B
I
C
L
C
B
A
Tuesday, March 5, 13
![Page 37: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/37.jpg)
A
09
I
C
A
V
B
10
R
E
B
I
C
L
C
B
A
Tuesday, March 5, 13
![Page 38: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/38.jpg)
A
09
I
C
A
V
B
10
R
E
B
I
C
L
C
B
A
Tuesday, March 5, 13
![Page 39: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/39.jpg)
After 5Hexawise
Tests
A
09
I
C
A
V
B
10
R
E
B
I
C
L
C
B
A
Tuesday, March 5, 13
![Page 40: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/40.jpg)
After 10Hexawise
Tests
A
09
I
C
A
V
B
10
R
E
B
I
C
L
C
B
A
Tuesday, March 5, 13
![Page 41: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/41.jpg)
After 13Hexawise
Tests
A
09
I
C
A
V
B
10
R
E
B
I
C
L
C
B
A
Tuesday, March 5, 13
![Page 42: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/42.jpg)
Now let’s compare coverage achieved by manual test case selection.
We’ll skip the “after 5 tests” and “after 10 tests” and go directly to “after 13 tests” (e.g., when Hexawise had already achieved 100%
coverage of all pairs of values).
Manual test case selection
Tuesday, March 5, 13
![Page 43: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/43.jpg)
After 13ManualTests
A
09
I
C
A
V
B
10
R
E
B
I
C
L
C
B
A
Manual test case selection
Tuesday, March 5, 13
![Page 44: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/44.jpg)
After 13ManualTests
A
09
I
C
A
V
B
10
R
E
B
I
C
L
C
B
A
This is worse than it might look at first...
Manual test case selection
Tuesday, March 5, 13
![Page 45: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/45.jpg)
In other words, when Hexawise tests had not only tested each test input but tested each test input in combination with every other test input in the plan at least once...
Manual test case selection
Tuesday, March 5, 13
![Page 46: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/46.jpg)
I
C
V
B
10
E
B
I
C
L
B
C
A
After 13ManualTests
AA
R
09
There were many, many pairs of values (in red) that the 13 manual tests had not tested together.
Manual test case selection
Tuesday, March 5, 13
![Page 47: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/47.jpg)
Even after the original plan’s 126 manually-created tests (or almost ten
times more test cases than the Hexawise test plan required)...
Manual test case selection
Tuesday, March 5, 13
![Page 48: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/48.jpg)
A
09
C
V
E
B
I
L
A
10
A
I
Nev
er te
sted
Never tested Never tested
Never tested
C
R
C
BAfter 126ManualTestsB
... there were four pairs of values that were never tested in the much longer original plan...
Manual test case selection
Tuesday, March 5, 13
![Page 49: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/49.jpg)
A
09
C
V
B
I
C
L
B
Tested 22 times
C
R
10
A
I
Tested 42 times
Tested 22 times
Tested 15 times
E
A
B
... and there were four pairs of values that were wastefully tested 15 or more times each.
Manual test case selection
Tuesday, March 5, 13
![Page 50: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/50.jpg)
A
09
C
V
B
I
C
L
B
Tested 22 times
C
After 126ManualTests
R
10
A
I
Tested 42 times
Tested 22 times
Tested 15 times
E
A
B
... and there were four pairs of values that were wastefully tested 15 or more times each.
Manual test case selection
Tuesday, March 5, 13
![Page 51: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/51.jpg)
A
09
C
V
B
I
C
L
B
Tested 22 times
C
R
10
A
I
Tested 42 times
Tested 22 times
Tested 15 times
E
A
B
Nev
er te
sted
Never tested Never tested
Never tested
These characteristics (wasteful repetition combined with missed gaps in coverage) are found in almost all
manually-selected test cases.Manual test
case selection
Tuesday, March 5, 13
![Page 52: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/52.jpg)
A
09
C
V
B
I
C
L
B
Tested 22 times
C
After 126ManualTests
R
10
A
I
Tested 42 times
Tested 22 times
Tested 15 times
E
A
B
Nev
er te
sted
Never tested Never tested
Never tested
These characteristics (wasteful repetition combined with missed gaps in coverage) are found in almost all
manually-selected test cases.Manual test
case selection
Tuesday, March 5, 13
![Page 53: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/53.jpg)
Without Hexawise With Hexawise
126 testsIncomplete coverageWasteful repetition
13 testsComplete coverage
Variation, not repetition
Tuesday, March 5, 13
![Page 54: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/54.jpg)
49
Source: Conservatively interpreted data from several dozen recent pilot projects. Time savings are often significantly larger than 40% and will almost always exceed 30%.
Faster Test Creation
Tuesday, March 5, 13
![Page 55: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/55.jpg)
50
More Defects Found / Hour
Source: Empirical study of average benefits 10 software testing projects published in IEEE Computer magazine in 2009: “Combinatorial Software Testing” Rick Kuhn, Raghu Kacker, Yu Lei, Justin Hunter. Results of individual projects will differ.
Tuesday, March 5, 13
![Page 56: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/56.jpg)
51
More Defects Found
Source: Empirical study of average benefits 10 software testing projects published in IEEE Computer magazine in 2009: “Combinatorial Software Testing” Rick Kuhn, Raghu Kacker, Yu Lei, Justin Hunter. Results of individual projects will differ.
Tuesday, March 5, 13
![Page 57: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/57.jpg)
52
Three FinalImplications
Tuesday, March 5, 13
![Page 58: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/58.jpg)
53
1. Bad software
quality can bring disaster to
anyone.
Tuesday, March 5, 13
![Page 59: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/59.jpg)
54
2.Smart, skilled,
empowered testers are essential.
Tuesday, March 5, 13
![Page 60: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/60.jpg)
55
3.Pairwise and
combinatorial testing helps test systems
Tuesday, March 5, 13
![Page 61: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/61.jpg)
55
3.Pairwise and
combinatorial testing helps test systems
BOTH more thoroughly
Tuesday, March 5, 13
![Page 62: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/62.jpg)
56
3.Pairwise and
combinatorial testing helps test systems
BOTH more thoroughly
Tuesday, March 5, 13
![Page 63: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/63.jpg)
56
3.Pairwise and
combinatorial testing helps test systems
BOTH more thoroughly AND more quickly.
Tuesday, March 5, 13
![Page 64: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/64.jpg)
57
1) Harry Robinson testing
2) Pairwise testing case studies
For more info, Google / Bing:
Tuesday, March 5, 13
![Page 65: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/65.jpg)
58
Tuesday, March 5, 13
![Page 66: Conquering the Largest Challenge of Software Testing: Too Much to Test & Not Enough Time to Test It All](https://reader033.vdocument.in/reader033/viewer/2022052505/5550fbbcb4c9057b478b4a5f/html5/thumbnails/66.jpg)
58
Thank You!
(BTW did this topic make your “Top 3” list?)
Tuesday, March 5, 13