testworksconf: experience exploratory testing
TRANSCRIPT
1
Experience Exploratory Testing
Maaike Brinkhof (@maaikees)Maaret Pyhäjärvi (@maaretp)
Exploratory testing is a systematic approach for discovering risks using rigorous analysis techniques coupled
with testing heuristics.
“[Exploratory Testing is] simultaneously designing and executing tests to learn about the system, using your insights from the last experiment to inform the next”
James Bach
4
Realizations on Nature of Testing
2016
16395±2
5
20DYNAMICALLY ADAPT FOR LIMITED BUDGET
OPPORTUNITY COST
6
16EXPECT THE UNEXPECTED
7
1639ROUTES ARE RELEVANT
NOT ALL BUGS ARE EQUAL
8
5±2TAKE NOTES
CREATE CHECKLISTS
9
Testing as Performance (Exploring) vs. Testing as Artifact Creation
10
What testing gives us
Uni
t Tes
ting
Exp
lora
tory
Tes
ting
SPECFEEDBACKREGRESSIONGRANULARITY
GUIDANCEUNDERSTANDING
MODELSSERENDIPITYTesting as
artifact creation
Testing as performance
11
EXPL
OR
ATO
RY T
ESTI
NG
SE
LF-M
AN
AG
EMEN
T
12
We’re at a stage in software industry where you can find 76 problems including a
showstopper in your first 4 hours of testing, ever. Imagine how the testing profession is
changing when that is no longer true?
13
Learning in LayersA Demonstration of Exploratory Testing
14
Teaching Exploratory Testing in a Mob
Mobbing Setup and Roles
Driver
Navigators
Designated Navigator
Facilitator
Mob Testing – Rules for the Exercise
No thinking as the driverYes, and…
Kindness, consideration and respect
The Test Target
JavaC#
GoPython
LuaPerl
NodeJS
18
1st Charter: from nothing to something
How would you learn something you’ve never seen before?
From Michael SahotaSTIC
KY
NO
TES
DO
NE
RIG
HT
Bring structure to ET: ChartersForm a mission before you testThink about the tools you want to useThink about what you’re hoping to find out
Example Charter:Explore ApprovalTests
By putting different variables in the verifyObject method
To discover if you get any unexpected behaviour
Examples from Elisabeth Hendrickson: Explore IT
22
2nd Charter: Get Specific
Explore ApprovalTestsBy putting different contents in the
verifyObject methodTo discover unexpected behaviour
23
Concepts
Tools
ET approaches, suggestions:SFDPO (James Bach)Sequences and interactions (Explore It!)Entities: CRUD, follow the data (Explore It!)States & transitions (Explore It!)Tours: super model tour, clubbing tour, couch potato, landmark tour, intellectual tour, FedEx tour Test Heuristics cheat sheet
More suggestions?
Suggested Tools:Browser: inspector, Postman, Bug Magnet, color picker, empty cache/cookies, SpectrumLogging/Proxy: Charles, FiddlerSecurity: OWASP ZAPPerformance/load: Gatling, JmeterGit
Any other suggestions?
Tools to support documentingRapid Reporter (mac/windows)Post It’sMicrosoft Exploratory Testing add-onTextEditScreen recordersA second (or 3rd, 4th…) pair of eyesCharles/Fiddler to record you API calls
More ideas?
27
Disposable test automation?
Programming tests for beyond regression?
Automation ideas beyond regressionFuzzingModel-based TestingMonitoring/analyticsStep to –toolsChaos monkeyState control with mocks/stubsPairing with automation ‘tour bus’Data generationMutation testing
28
Main characteristics of ET Self-directednessFreedomResponsibilityLearningWorking with your own biasesImproving yourself as a testerContextual
30
Maaret PyhäjärviEmail: [email protected]: @maaretpWeb: maaretp.comBlog: visible-quality.blogspot.fi
(please connect with me through Twitter or LinkedIn)
Maaike BrinkhofEmail: [email protected]: @maaikeesWeb: maaikebrinkhof.nlBlog: maaikebrinkhof.nl
32
Extra slides beyond this one
Example ChartersExplore input fieldsWith injection attacksTo discover security vulnerabilities
Explore editing profilesWith invalid usernamesTo discover if there are any instances where username constraints are not enforced
Example Charters (2)Explore editing profilesWith invalid usernamesTo discover if there are any instances where username constraints are not enforced
ET DefinitionCem Kaner:“Exploratory testing is a style of software testing that emphasizes the personal freedom and responsibilityof the individual tester to continually optimize the value of her work by treating test-related learning,test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”
James Bach:“Simultaneously designing and executing tests to learn about the system, using your insights from the last experiment to inform the next”
Testing vs Checking debateChecking: tests you design in advance, can be automated, can be scaled, modeled, etc.
Testing: only a human can do testing, it involves judgement on the fly, skill, sharp decision making. You cannot test everything! So test wisely
Context Driven PrinciplesThe value of any practice depends on its context.There are good practices in context, but there are no best practices.People, working together, are the most important part of any project’s context.Projects unfold over time in ways that are often not predictable.The product is a solution. If the problem isn’t solved, the product doesn’t work.Good software testing is a challenging intellectual process.Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.
Agile ManifestoIndividuals and interactions over processes and toolsWorking software over comprehensive documentationCustomer collaboration over contract negotiationResponding to change over following a plan
That is, while there is value in the items onthe right, we value the items on the left more.
That's where the power of exploratory testing comes in: the richness of this process is only limited by the breadth and depth of our imagination and our emerging insights into the nature of the product under test.