exploratory testing in practice

Exploratory Testing

in Practice

Elisabeth Hendrickson Blog: testobsessed.com Twitter: @testobsessed

Slides Last updated June 22, 2012

Two Sides of Testing

Tested =

+ Checked Explored

That’s Nice…

…But we Believe in our

comprehensive, repeatable regression

tests.

Incidental Correctness

Given a series of 1’s and 0’s,

Compute the corresponding

decimal number

x

6

5

7

0

011

101

111

000

3

5

7

0 Result Input Expected

Missed Requirements

Hey, Where’s the motor for this

remote control car?

Motor?!? You never

said anything about a motor!

Real World Users Happen

You assume they’re

reasonable people

who will do

reasonable things.

They aren’t and

they won’t.

So What Is Exploratory Testing?

a systematic approach for

discovering risks

using rigorous

analysis techniques

coupled with

testing heuristics.

OK…But What Do You Actually Do?

Simultaneously…

…learning about the Target

…While designing tests

…And executing Them Immediately

using feedback from the last Little Experiment to inform the next

(Term coined by Cem Kaner. Definition refined by James Bach. Others have contributed to ET including

James Lyndsay, Jonathan Kohl, Michael Bolton.)

Experiment

Next, let’s try interrupting the

transaction here.

So it handled all the variations

on disconnected networks we

could think of…

Discover Things You Can Vary

11

Use State Models

Update System

Check Network

Idle

Time to connect

System Updated

No Network

Get Updates

Network Avail

No Updates

Content Downloaded

Use Data Models

Customer

Orders

Vendor

Products

Has 0 or More

Has 0 or More

Has 1 Or More

Use Deployment Diagrams

Local Server Farm

Agent AWS

Use Heuristics

Counts: 0, 1, Many

See the Test Heuristics Cheatsheet for more http://bit.ly/testcheatsheet

Position: Beginning, Middle, End

Selection: Some, None, All

Violate Domain-Specific Rules:

Age -1 IP address 999.999.999.999

Goldilocks: too big, too small,

just right

Interrupt States

Delete Dependent Data

In Flight Feedback at #AU2H

Jon Bach

(Exploring)

Matt Barcomb

(Coding)

Result: Feedback on Process AND Product

I’ll add things like that to the

acceptance criteria.

We need to think about that kind of

thing in future designs too.

We’ve automated the regression

test when we fixed the bug.

Pitfall: Going Off in the Weeds

it doesn’t work on 15 year old laptops running Windows 3.1!

Uh…

Why Would We

Care?

Focus with Charters

Explore (Area/feature)

With (Resources,

conditions, Dependency,

etc.)

to discover (information)

Structure with Time Boxed Sessions

Session

Session

Session

Session

Session

Session

Explore login in combination with bookmarks & history to discover information about interactions.

Explore the site with various browsers and settings to discover risks related to unsupported configurations.

Explore error handling using data type attacks to find problems related to data validation.

Charters Sessions

Explore Early, Explore Often

Exploratory Testing on Agile Projects

Within an Iteration

Automated Unit Tests

Automated Acceptance

Tests

Exploratory Testing

Define “done”, represent executable

requirements

Drive design, represent executable

specifications

Discover risks EARLY