testworksconf: experience exploratory testing

1

Experience Exploratory Testing

Maaike Brinkhof (@maaikees)Maaret Pyhäjärvi (@maaretp)

Exploratory testing is a systematic approach for discovering risks using rigorous analysis techniques coupled

with testing heuristics.

“[Exploratory Testing is] simultaneously designing and executing tests to learn about the system, using your insights from the last experiment to inform the next”

James Bach

4

Realizations on Nature of Testing

2016

16395±2

5

20DYNAMICALLY ADAPT FOR LIMITED BUDGET

OPPORTUNITY COST

6

16EXPECT THE UNEXPECTED

7

1639ROUTES ARE RELEVANT

NOT ALL BUGS ARE EQUAL

8

5±2TAKE NOTES

CREATE CHECKLISTS

9

Testing as Performance (Exploring) vs. Testing as Artifact Creation

10

What testing gives us

Uni

t Tes

ting

Exp

lora

tory

Tes

ting

SPECFEEDBACKREGRESSIONGRANULARITY

GUIDANCEUNDERSTANDING

MODELSSERENDIPITYTesting as

artifact creation

Testing as performance

11

EXPL

OR

ATO

RY T

ESTI

NG

SE

LF-M

AN

AG

EMEN

T

12

We’re at a stage in software industry where you can find 76 problems including a

showstopper in your first 4 hours of testing, ever. Imagine how the testing profession is

changing when that is no longer true?

13

Learning in LayersA Demonstration of Exploratory Testing

14

Teaching Exploratory Testing in a Mob

Mobbing Setup and Roles

Driver

Navigators

Designated Navigator

Facilitator

Mob Testing – Rules for the Exercise

No thinking as the driverYes, and…

Kindness, consideration and respect

The Test Target

JavaC#

GoPython

LuaPerl

NodeJS

18

1st Charter: from nothing to something

How would you learn something you’ve never seen before?

From Michael SahotaSTIC

KY

NO

TES

DO

NE

RIG

HT

Bring structure to ET: ChartersForm a mission before you testThink about the tools you want to useThink about what you’re hoping to find out

Example Charter:Explore ApprovalTests

By putting different variables in the verifyObject method

To discover if you get any unexpected behaviour

Examples from Elisabeth Hendrickson: Explore IT

22

2nd Charter: Get Specific

Explore ApprovalTestsBy putting different contents in the

verifyObject methodTo discover unexpected behaviour

23

Concepts

Tools

ET approaches, suggestions:SFDPO (James Bach)Sequences and interactions (Explore It!)Entities: CRUD, follow the data (Explore It!)States & transitions (Explore It!)Tours: super model tour, clubbing tour, couch potato, landmark tour, intellectual tour, FedEx tour Test Heuristics cheat sheet

More suggestions?

Suggested Tools:Browser: inspector, Postman, Bug Magnet, color picker, empty cache/cookies, SpectrumLogging/Proxy: Charles, FiddlerSecurity: OWASP ZAPPerformance/load: Gatling, JmeterGit

Any other suggestions?

Tools to support documentingRapid Reporter (mac/windows)Post It’sMicrosoft Exploratory Testing add-onTextEditScreen recordersA second (or 3rd, 4th…) pair of eyesCharles/Fiddler to record you API calls

More ideas?

27

Disposable test automation?

Programming tests for beyond regression?

Automation ideas beyond regressionFuzzingModel-based TestingMonitoring/analyticsStep to –toolsChaos monkeyState control with mocks/stubsPairing with automation ‘tour bus’Data generationMutation testing

28

Main characteristics of ET Self-directednessFreedomResponsibilityLearningWorking with your own biasesImproving yourself as a testerContextual

30

Maaret PyhäjärviEmail: maaret@iki.fiTwitter: @maaretpWeb: maaretp.comBlog: visible-quality.blogspot.fi

(please connect with me through Twitter or LinkedIn)

Maaike BrinkhofEmail: mbrinkhof@xebia.comTwitter: @maaikeesWeb: maaikebrinkhof.nlBlog: maaikebrinkhof.nl

32

Extra slides beyond this one

Example ChartersExplore input fieldsWith injection attacksTo discover security vulnerabilities

Explore editing profilesWith invalid usernamesTo discover if there are any instances where username constraints are not enforced

Example Charters (2)Explore editing profilesWith invalid usernamesTo discover if there are any instances where username constraints are not enforced

ET DefinitionCem Kaner:“Exploratory testing is a style of software testing that emphasizes the personal freedom and responsibilityof the individual tester to continually optimize the value of her work by treating test-related learning,test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project.”

James Bach:“Simultaneously designing and executing tests to learn about the system, using your insights from the last experiment to inform the next”

Testing vs Checking debateChecking: tests you design in advance, can be automated, can be scaled, modeled, etc.

Testing: only a human can do testing, it involves judgement on the fly, skill, sharp decision making. You cannot test everything! So test wisely

Context Driven PrinciplesThe value of any practice depends on its context.There are good practices in context, but there are no best practices.People, working together, are the most important part of any project’s context.Projects unfold over time in ways that are often not predictable.The product is a solution. If the problem isn’t solved, the product doesn’t work.Good software testing is a challenging intellectual process.Only through judgment and skill, exercised cooperatively throughout the entire project, are we able to do the right things at the right times to effectively test our products.

Agile ManifestoIndividuals and interactions over processes and toolsWorking software over comprehensive documentationCustomer collaboration over contract negotiationResponding to change over following a plan

That is, while there is value in the items onthe right, we value the items on the left more.

That's where the power of exploratory testing comes in: the richness of this process is only limited by the breadth and depth of our imagination and our emerging insights into the nature of the product under test.