exploratory testing: make it part of your test strategy
TRANSCRIPT
Exploratory Testing: Make It Part of Your Test Strategy
Kevin DunneVP, Strategy and Business Development
QASymphony
Agenda
1. What is exploratory testing (ET)?
2. What is NOT ET?
3. Why should we do ET?
4. How can I integrate ET with BDD?
5. ET best practices – do’s and don’ts
What is exploratory testing?
1. Parallel test planning, test design, and test execution
2. Specific yet flexible
3. Aligned towards investigation of potential opportunities
4. Values depth and attention to detail during testing
5. Fosters knowledge sharing and awareness
Parallel Planning, Design & Execution
Unlike traditional testing techniques, planning, design, and execution happen concurrently, allowing efficiencies of time as well as flexibility in approachPlan
Design
Execute
Report
Plan
Design
Execute
Report
Specific yet Flexible
Exploratory testing provides a specific lens through which to perform testing – whether that be a user persona, functionality, criteria (i.e. localization), etc.
However, it allows testers to use the tool as an end user would, not necessarily as the product owner envisioned it
Manual Scripted Testing
I tested the application as the script prescribed
Exploratory Testing
I tested the application as the end user would
Investigating Opportunities
Exploratory testing rewards testers who identify unknown areas of “opportunity” within the application, as they are essential in maintaining a backlog of future test charters
Manual Scripted Testing Exploratory Testing
Knowledge Sharing
Exploratory testing relies on knowledge sharing to reach full potential – developing testers who understand the impact of more areas of the application allows them to identify more areas of risk and opportunity
Plan
Design
Execute
Report Transfer Learning
Example Questions to Ask
• Have you seen this before?
• What am I not considering?
• Why would someone do this?
• How would you have tested this?
What is Not Exploratory Testing
1. Exploratory testing is NOT unstructured testing
2. Exploratory testing is NOT the only form of testing
3. Exploratory testing is NOT throwaway work
4. Exploratory testing is NOT impossible in a regulated environment
ET is NOT Unstructured Testing
While exploratory testing allows for flexibility in the exact path of the application that is tested, it is NOT unstructured, in that it still contains parameters such as:
1. A goal of the exploration
2. A log of the activity performed
3. A lens through which the testing is performed (i.e. a user persona)
Performing exploratory testing without involving some parameters such as the above allows a greater risk of unsuccessful implementation of exploratory testing
ET is NOT the Only Form of Testing
Exploratory testing is best suited as a complement to automated and manual scripted test cases. It can feed these types of testing to create greater depth in testing, and also to identify any potential gaps in coverage.
Potential New Feature Testing Cycle
Code
Developer
Unit Test
Exploratory Testing
Manual Scripted Test
Automatio
n Regression Test
Exploratory Testing
Feature “Delivered”
“Let’s make sure this is worth writing
scripts against yet”
“Let’s make sure were still testing all
aspects of this”
ET is NOT Throwaway Work
Exploratory testing does NOT need to be extra work done on top of other testing methods – it can count on its own towards testing progress and coverage if properly accounted for. Some of the necessary information needed to manage it:
1. Charter2. Session Sheet3. Oral report4. Debrief5. Data Files6. Logs
"Any testing approach is manageable when you choose to manage it.” – Michael Boltonhttp://www.developsense.com/blog/2010/01/exploratory-testing-is-accountable/
ET is NOT Impossible in a Regulated Environment
Contrary to rumor and popular belief, exploratory testing is not only allowed in most regulated environments, it is also essential.
Why Should we do ET?
A 2007 controlled study found that:
• Testing with test cases vs. exploratory testing take almost 7 times longer, due to the amount of time needed to write the tests and report results on them
• Testing with test cases vs. exploratory testing finds more defects, and does not miss many (if any) critical or severe defects in comparison to test case testing
• Testing with test cases causes more false defect reports vs. exploratory testing
Study link: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.167.3696&rep=rep1&type=pdf
Adding ET to your Test Strategy
1. Paired Testing – get quick feedback from product experts early in the development cycle
2. Team Based ET – rally the entire team around product learning and exploration
3. UAT – focus valuable SME time on value added activity rather than documentation
4. Beta Testing – maximize the returns of beta testers, minimize duplication and uncovered areas
5. Replacing Traditional Testing – shift “stale” manual scripted tests to drive net new exploration and feedback
Traditional Problems in Siloed Testing
Unsure if problem is code or requirements• Testing is removed from development and requirements
management
Bottlenecks in handoffs between test and dev• Submitting defects, assisting in defect reproduction,
notifying tester of defect ready for retest
Too Much Time Passing Between Dev and test• Developer is notified of bugs in their code days after testing
has been completed
Benefits of Paired Testing
Close integration of development and test• Minimize the risk of miscommunication and incorrect
interpretation of requirements
Immediate feedback on new code• Code is tested right away, meaning developers have less
time to continue to build on bad code that will need to be rewritten
Fix Problems Immediately and Move On• Reduce the number of defects opened, decreasing the
number of issues that would make it into production
Team Based ET
Exploratory testing is something that the whole team can benefit from:
1. Easy to Learn – don’t need to be proficient in many tools, languages, etc.
2. Benefits from multiple perspectives and viewpoints3. Quick to start up and scale – reduced overhead to get process
set up and running
NOTE: just because some team members are not experienced testers, that does not mean that you throw the basics of exploratory testing out the window!
UAT with ET
UAT Challenge ET Benefit
UATer’s are unfamiliar test case syntax and need continual clarification
Allow UATer’s to perform the business flows they know well without test scripts
UATer’s are not trained on test case management, automation tools, etc.
Focus UATer’s time on learning how to document proper defects, reduce time to ramp
UATer’s have a shorter attention span – they are not used to testing 6-8 hrs. per day
Allow UATer’s to veer off the rails from time to time and investigate areas of interest
UATer’s have a short period of time in which to provide feedback
Ensure that as much of the UATer’s time as possible is dedicated to ET
Beta Testing Challenges Solved Through ET
Problem #1 - Even the most efficient beta testing shops rarely get feedback from more than 30% of their users – meaning that 70% of the beta testers provide no feedback
Solution – provide specific ET charters to beta testers. Charters will keep the testers focused on key objectives and will drive accountability that will increase participation
Problem #2 – There is a segment of use cases are either under or over tested, leaving bugs undiscovered in production
Solution – prioritize particular features and use intelligent assignment of the specific charters to make sure adequate coverage is planned across appropriate environments and devices
Traditional Beta vs. ET Beta Testing
Traditional Beta ET BetaWhere should I focus testing? At the users discretion, where
they’d likeAccording to their assigned charter
How many times will this feature get tested?
We don’t really know As many times as it is assigned out to users
Will this feature get tested on all environments?
We don’t really know We will assign it to environments needing coverage
Are testers focusing their efforts on new features being released?
We don’t really know Yes, assuming we assign them to work in those particular areas
Replace Traditional Testing with ET
Workplace choice improves the employee experience, and adding exploratory testing to the mix allows testers to have choice many times per day
Replace Traditional Testing with ET
There are typically two strategies, which can be used in conjunction to begin replacing manual scripted tests with exploratory ones:
1. Look for tests that have resulted in the lowest failure rates, lowest defect detection rates, or both. In priority order, transfer these tests to exploratory test charters, and monitor the defect detection rates from the transition.
2. Look for tests that take the longest to run, and are run the most frequently. In priority, transfer these tests to exploratory charters, monitor the time per execution, and ensure that defect detection rates stay constant or improve.
How to best structure our exploratory testing?
Introducing exploratory testing within a framework will greatly increase the odds of success, and will reduce fear and uncertainty among the practitioners as well as executives.
Session Based Test Management is a popular framework for this, as it tracks all the important data on testing:
More info on SBTM: http://www.satisfice.com/articles/sbtm.pdf
• Session charter (includes a mission statement, and areas to be tested)
• Testers involved Date and time executed• Task breakdown
• Data files• Test notes• Issues • Bugs
Resources/Thought Leaders
• James Bach - http://www.satisfice.com/• Jonathan Bach - https://jonbox.wordpress.com/• Michael Bolton - http://www.developsense.com/• Paul Holland -
http://testingthoughts.com/blog/author/testthought• Keith Klain - http://qualityremarks.com/• Brian Osman - https://bjosman.wordpress.com/
Questions?Kevin Dunne
[email protected]: @kevindunneQA
Linkedin: www.linkedin.com/in/kevindunneQABlog: http://www.qasymphony.com/blog/