@ingophilipp - theory and practice of software testing ... · @ingophilipp ‹› exploration...

Rediscover Exploratory Testing

@IngoPhilipp

@IngoPhilipp

‹› Think automation first. Automation should be accessible for everyone. One robot for every person.@IngoPhilipp

Just as glasses do not see, tools do not test.Glasses help us to see, tools help us to test

James Bach

‹› Glasses don't do the seeing, tools don't do the testing.@IngoPhilipp

Testing is no more about test casesthan astronomy is about telescopes

Ingo Philipp

‹› A recipe (test case) is not cooking (testing). We automate test cases, not testing.

∷ Inspired by Edsger Wybe Dijkstra & Michael Bolton

@IngoPhilipp

If we’re testing badly, then automation canhelp us to test badly at an accelerated rate

Michael Bolton

‹› Automation accelerates, intensifies, amplifies, often simplifies, and sometimes even enables testing.

@vertaontest ‹› If you automate chaos, all you get is faster chaos.

@IngoPhilipp

∷ Inspired by Brendan Leonard

Testing? There's only automation!We automate everything

OK, there's testing, and testingis probably more than just automation.

We're just not convinced that we really need it

Oops

Damn!

‹› Confusing testing with automation is a dangerous thing.@IngoPhilipp

Test Case

@IngoPhilipp

PassingTest Case

@IngoPhilipp

PassingTest Case

@iamdevloper

@IngoPhilipp

© BBC

‹› Software development is a lot like wresting in the mud with a pig.@IngoPhilipp

Testing is exactly likewashing a pig. Because

it’s messy. It has no rules. Noclear beginning, middle, or end.

It’s kind of a pain in the ass, and when you’re done you’re

not sure if the pig is really clean or even why you were washing

a pig in the first place.

∷ Inspired by Luke Sullivan © BBC

@IngoPhilipp






∷ Inspired by Luke Sullivan

‹› We actually wash a performing pig.@IngoPhilipp







That’s what we can know‹› We actually wash a performing pig.@IngoPhilipp







That’s what we can know

That’s what we know

That’swhat wecheck

Problem

Knowledge Gap

© Richard Bradshaw@IngoPhilipp

∷ Cem Kaner

The purpose of testing is toclose the knowledge gap

The goal is information,not gratuitous automation

Testing is and always will bea search for information




Problem

Knowledge Gap





Problem

Knowledge Gap

The information soughtoften, but not always, requires

automation

We're an information service,not an automation service

We enable people to makebetter decisions based on the

information we provide





Problem

Knowledge Gap


CheckedMachine & Human Checking

ExploredHuman Exploration & Experimentation

TestedThorough Testing

+

=

∷ Elisabeth Hendrickson ‹› Testing Equation

‹› The coin of testing has two sides.




Problem

Knowledge Gap

© Richard Bradshaw

Checking« Requires Processing »

∷ Bolton ‹› Kaner ‹› Bach

Evaluate a product by applying algorithmicdecision rules to specific observations of a product

Change DetectorDoes this assertion pass or fail?

@IngoPhilipp


∷ Bolton ‹› Kaner ‹› Bach

Evaluate a product by applying algorithmicdecision rules to specific observations of a product

Change Detector

Exploring« Requires Thinking »

Problem Detector

Evaluate a product by learning about itthrough exploration and experimentation

Does this assertion pass or fail? Is there a problem here?

@IngoPhilipp


∷ Rich Rogers

Evaluate through

InstructionsPay attention to

Deviations

Create

Test CasesFollow

Procedure

Examine

RequirementsFactory

Process

Change Detectors ‹› Monitor Known Risks


Problem Detector

Evaluate a product by learning about itthrough exploration and experimentation

Is there a problem here?

@IngoPhilipp


∷ Rich Rogers

Evaluate through


Deviations

Create

Test CasesFollow

Procedure

Examine

RequirementsFactory

Process

Change Detectors ‹› Monitor Known Risks

Investigate through

ExperimentsPay attention to

Oracles

Create

Test IdeasFollow

Clues

Examine

RisksAdaptive

Investigation

Problem Detectors ‹› Analyze Potential Risks


@IngoPhilipp


∷ Rich Rogers

Evaluate through


Deviations

Create

Test CasesFollow

Procedure

Examine

RequirementsFactory

Process

Me

ch

an

ica

l P

roce

ssChange Detectors ‹› Monitor Known Risks

Investigate through

ExperimentsPay attention to

Oracles

Create

Test IdeasFollow

Clues

Examine

RisksAdaptive

Investigation

Problem Detectors ‹› Analyze Potential RisksC

rea

tive

Pro

cess


Robots Investigators

@IngoPhilipp

Behind a test case, there's a test. Behind a test, there's atest idea. Behind a test idea, there's a human tester

Ingo Philipp

‹› If your test case is good-for-nothing, you are probably good-for-nothing.

∷ Your test success is defined by the quality of your test ideas, not by the number of your test cases.

@IngoPhilipp

Automated Checking is a minuscule doton a minuscule dot in the space of testing

Ingo Philipp

‹› Exploration isn’t something special in testing, it is central to testing, and so all true testing is exploratory.@IngoPhilipp

∷ Michael Bolton ‹› Checking is a guest in the house of testing, exploration is a permanent resident.

Exploratory testing is not a talent,it’s a set of skills that can be learnt

Ingo Philipp

‹› The beautiful thing about learning exploratory testing is that no one can take it away from you.

∷ Inspired by Edward de Bono

@IngoPhilipp

Technique

Provides Systematic Procedure

Approach

Provides Orientation

@IngoPhilipp

1

2

3

4

5

Technique


@IngoPhilipp

Session-Based TestingStructure exploratory testing to allow large-scale implementations

2

3

4

5

Technique


∷ Jonathan Bach

@IngoPhilipp


2

3

4

5Straightjacketed Imagination

@IngoPhilipp


2

3

4

5

Session

CharteredUninterrupted

Reviewable

∷ Jonathan Bach

@IngoPhilipp


2

3

4

5

Timebox

∷ Jonathan Bach

@IngoPhilipp


2

3

4

5

Charter

∷ Jonathan Bach

@IngoPhilipp


Requirements-Based TestingLimit the scope to make it manageable

3

4

5

Scope

@IngoPhilipp



3

4

5Verification

Does our software meet the specification?

ValidationDoes our software meets the user's needs?

ProductIdea

ActualProduct

ProductDescription

You

@IngoPhilipp

∷ Heads up! ‹› Think of falsification instead of verification.

BDD is the art of usingexamples in conversations

to illustrate behavior

Liz Keogh

‹› Use BDD to create a shared understanding.

Nobody (e.g. product owner)can make you (e.g. tester) think

& feel as deeply as they do about software. Be aware of this fact

to explore this gap.

∷ Inspired by Daniel Saint



3

4

5

@IngoPhilipp

HavingConversations

is more important than

CapturingConversations

is more important than

AutomatingConversations

∷ Liz Keogh ‹› CAST 2018

‹› BDD is about understanding, not test case automation.







3

4

5

@IngoPhilipp







3

4

5

What theycan put into

words

What theysay to their

teams

Whatproductownersthink

What you usuallyunderstand

∷ arcfly.blogspot.com

‹› We don't just explore the unknown in our software.@IngoPhilipp



3

4

5

R » RecentWhat parts of the product changed recently?

C » CoreWhat critical parts of the product must continue to work?

R » RiskyWhat parts of the product are inherently risky?

C » ConfigurationWhat parts of the product depend on environment settings?

R » RepairedWhat parts of the product changed to address defects?

C » ChronicWhat parts of the product chronically break?

∷ Karen Johnson

@IngoPhilipp



3

4

5

S » StructureTest what the product is made of.

F » FunctionTest what the product does.

D » DataTest what the product processes.

P » PlatformTest what the product depends upon.

O » OperationsTest how the product is used.

T » TimeTest how the product is affected by time.

∷ James Bach

@IngoPhilipp



Tour-Based TestingSet concrete goals to provide a clear focus

5

4Goals

MuseumTour

SupermodelTour

FedExTour

SaboteurTour

Couch PotatoTour

MoneyTour

∷ James Whittaker

@IngoPhilipp




5

4@speed

QualityQuality is value to some person

∷ Jerry Weinberg

@IngoPhilipp




5

4

Quality is inherently subjective

Different stakeholderswill perceive the same product ashaving different levels of quality

We must look for differentthings for different stakeholders

We must diversify testing

∷ Jerry Weinberg

@IngoPhilipp




Polychrome TestingExplore the product from different viewpoints to diversify testing

5

@IngoPhilipp

Ideas





Scenario-Based TestingCapture each test idea to make it reviewable

@IngoPhilipp

Testing is not so much a thing you do,it’s far more a way you think

Michael Bolton

‹› Testing is enactment of critical thinking about software to help people make better decisions.@IngoPhilipp

QuestionsThe show is over. It’s your turn.

?

@IngoPhilipp

Before Checking‹› Before the check happens ‹›

After Checking‹› After the check happened ‹›

Perform

Check

You must learn about your softwareThat takes testing skill

You must understand your softwareThat takes testing skill

You must recognize, identify & prioritize a risk That takes testing skill

You must translate that risk into a test ideaThat takes testing skill

You must express a test idea yes-or-no questionThat takes testing & programming skill

You must turn the question into codeThat takes programming skill

You must find a trigger to launch the check That takes testing skill

You must encode the trigger to automate it That takes programming skill

You must read the bit That takes programming skill

You must aggregate bitsThat takes programming skill

You must design a reportThat takes design & testing skill

You must encode the reportThat takes programming skill

You must observe the reportThat takes testing skill

You must determine meaningThat takes testing & product mgmt. skill

You must judge significanceThat takes testing & project mgmt. skill

You must respondThat takes at least one of several of the skills above

‹› Rapid Software Testing ‹›

@IngoPhilipp

Learning

Exploration

Experimentation

Modelling

Studying

Drawing inferences

Questioning facts

Assessing risks

Prioritizing risks

Analyzing coverage metrics

Making decisions

Designing test labs

Preparing test labs

Advocating testability

Applying judgement

Applying critical thinking

Applying emotional thinking

Applying logical thinking

Developing test code

Here's a substitution of testing

Can testing beautomated?

?

∷ Michael Bolton ‹› There's no such thing as automated testing.

@IngoPhilipp

‹› The number one testing tool is not the computer, it is still the human brain ‹› Jerry Weinberg.

∷ Picture by Shelly Palmer

@IngoPhilipp

@ingophilipp - theory and practice of software testing ... · @ingophilipp ‹› exploration...

Documents