just-in-time testing workshop - let's...

AmiBug.Com, Inc.May 7, 2012 © Robert Sabourin, 2012 Slide 1

Just-In-Time TestingWorkshop

Robert Sabourin

AmiBug.Com, Inc.

Montreal, [email protected]


Overview

WelcomeSome PhilosophyBe Pre[paredTesting IdeasTest Triage


Just In Time Testing

• Robert Sabourin , Software Evangelist

• President

• AmiBug.Com Inc.

• Montreal, Quebec, Canada

• [email protected]

• www.amibugshare.com


Just-In-Time Testing

Welcome



• Pain points?– What hurts?

– How Much?



Just In Time For ________?



Some Philosophy


Fundamental Question

• How do you know when you are finished?


Crosby on Quality

• “Quality is defined as conformance to requirements”

• “Quality is not a measure of GOODNESS”– Phil B. Crosby, Quality is Free


“Quality is fitness for use”

Joseph Juran

Quality Control Handbook


Gerald M. Weinberg

“Quality is value to some person”

Exploring Requirements Quality Before Design

Dorset House


Edsger W. Dijkstra

• “Program testing can be used to show the presence of bugs, but never to show their absence”

AmiBug.Com, Inc.

Quiz

• Application screensare selected with three controls:(a) has 5 options

(b) has 6 options

(c) has 2 options

• How many screens can a user choose?

May 7, 2012 © Robert Sabourin, 2012 Slide 13

AmiBug.Com, Inc.

Quiz

• Total Combinations = 6 x 5 x 2 = 60

• To exercise each combination once a total of 60 tests would be required.


AmiBug.Com, Inc.

• How many tests would be required to exercise all possible screens in every possible order?.

May 7, 2012 © Robert Sabourin, 2012

Slide 15

Quiz

AmiBug.Com, Inc.

• To exercise all screens in every possible order would require

60! Test cases

60! = 60 x 59 x 58 x ... 3 x 2 x 1

60! ≈ 8.32 x 10**81


Pop Quiz

Slide 16

AmiBug.Com, Inc.

Pop Quiz

From 7.0 × 10**79

To 1.5 × 10**82


How many atoms are in the observable universe?

Slide 17


Control Flow Testing

• Model flow– Create control flow

diagram– Find basis paths

• Minimal set of transactions• Exercise at least once

– Each step– Each decision



– Minimal basis paths

• N – number of nodes

• E – number of edges

• P – number of basis paths

• P = E – N + 2

• McCabe Cyclomatic Complexity

B

A

C D

F G

E

H

I

N=9E=10

P=10-9+2P=3



– Finding basis paths:

1. Start with a typical baseline2. Flip first decision keep rest as

similar as possible3. Continue flipping decisions on

baseline4. After all decisions on baseline

have been flipped continue on next path

5. Stop when all paths have been exhausted

B

A

C D

F G

E

H

I

N=9E=10

P=10-9+2P=3

AmiBug.Com, Inc.

ST

Di

M

R

N

Q

P

L A

B

C

D

E

FG

I

J

K

O

H

S

T

U

V W

X

ZY

ACAD

AEAF AA

AG AH AJ ABAMAL

AIAK

AN

AO APAQ



Control Flow Diagram to Identify Basis PathsBasis Paths

p01 Start Dispatch L M N P Q Rp02 Start Dispatch X Y AA AB Rp03 Start Dispatch AC AF AJ AI AK Rp04 Start Dispatch AD AE AF AJ AI AK Rp05 Start Dispatch O P Q Rp06 Start Dispatch K P Q Rp07 Start Dispatch I J C D H P Q Rp08 Start Dispatch A B C D H P Q Rp09 Start Dispatch L M N P S T Q Rp10 Start Dispatch L M N P S T U Rp11 Start Dispatch L M N P S T U V W Rp12 Start Dispatch X Z AA AB Rp13 Start Dispatch X Z AA AM Rp14 Start Dispatch X Z AA AM AN AP AQ Rp15 Start Dispatch X Z AA AM AN AO AQ Rp16 Start Dispatch AC AF AG Rp17 Start Dispatch AC AF AH Rp18 Start Dispatch AC AF AJ AL AM Rp19 Start Dispatch AC AF AJ AL Rp20 Start Dispatch I J C D E F Rp21 Start Dispatch I J C D E G Rp22 Start Dispatch A B R

Nodes 45Edges 65Complexity 22 E-N+2


Purpose of Testing

• Common definition:– To find bugs before our customers do!

• Broader definition:– The role of testing is to provide objective input to

facilitate business decisions!

– Keeps stakeholders aware of all issues or concerns that relate to shipping a product!


Bug Defined

• To make our job more fun, whenever we have a concern with software, we call it a “bug”.


• It’s all about people! (and the occasional bug too)




Context Drivers


Context Drivers - BTO

• Business– Value– To whom?– Why?

• Technology– Solutions

• Organization– Corporate Structure– Team Structure– Roles and Responsibilities



Context Listeners

• Find Sources

• Monitor Drivers

• Anticipate Change

• React



Get Ready, Get Set,Cause here it comes


Turbulence

(.wav)



Unprepared




Sharpen Testing Skills

• Thinker

• Detective

• Reporter

• Diplomat

• Negotiator

• Cheer Leader

• Pragmatist


Philosophy

We have precious little time to run tests!

We must always be prepared!


Time



Test Triage


"No! Try not, Do. Or do not.

There is no try."

YodaP

lan

to s

uppo

rt c

hang

e


Testing Ideas

• Collect all testing ideas you can find!– List

– Sort

– Organize

– Shuffle

Pla

n to

sup

port

cha

nge


Testing Ideas

• How to find them?– Does system do what it is suppose to do?

– Does the system do things it is not supposed to?

– How can the system break?

– How does the system react to it’s environment?

– What characteristics must the system have?

– Why have similar systems failed?

– How have previous projects failed?

Pla

n to

sup

port

cha

nge


Testing Ideas

• Collect testing ideas

• From testing ideas build a series of testing objectives– Each can be assigned as

work to testers

– Each can include all, part of, or multiple testing ideas

Cap

ture

tes

ting

idea

s


Testing Ideas

• I often use Index Cards– Unique id

– One testing idea per card

– Colour indicates source

– Shuffled and reviewed

– Organized and reorganized

– Sorted, grouped, prioritized and collected

Cap

ture

tes

ting

idea

s


Test Idea Sources• Capabilities• Failure Modes• Quality Factors• Usage Scenarios• Creative Ideas• States• Data• Environments• White Box• Taxonomies

Cap

ture

tes

ting

idea

s


Testing Ideas

• Investigative approaches– We become truffle

snorting pigs and try to find useful information in all evidence we discover

– We can even get good ideas from out of date sources

Cap

ture

tes

ting

idea

s


Testing Ideas• Capabilities

– Use cases– Functional

requirements– Documented

requirements– Implicit requirements

Cap

ture

tes

ting

idea

s


Testing Ideas

• Failure Modes– What can break?– Reaction to invalid input?– How does software

behave in constrained environment?

• Memory• Disk Space• Network Bandwidth• CPU capacity• Shared resources

– Stress, Load, Volume

Cap

ture

tes

ting

idea

s


Ada

ptab

ility

Acc

essi

bilit

y

Aud

itabi

lity

Ava

ilabi

lity

Con

tinui

ty

Dep

enda

bilit

y

Exp

anda

bilit

y

Fun

ctio

nalit

y

Inte

grity

Inte

rope

rabi

lity

Mai

ntai

nabi

lity

Ope

rabi

lity

Por

tabi

lity

Rel

iabi

lity

Re-

usab

ility

Sca

labi

lity

Sec

urity

Ser

vice

abili

ty

Test

abili

ty

Usa

bilit

y

Application service providerAutomatic content generatorCustomized accessDatabase accessDeliveryDocument accessFile sharingInformationalInteractiveTransaction orientedUser-provided contentWorkflow oriented

High FocusMedium Focus

Low Focus

Quality Factors ImportanceDifferent Application Types

Cap

ture

tes

ting

idea

s


Testing Ideas

• Usage Scenarios– Identify classes of

users

– Identify how users will use system

– Describe scenarios

– Use Story board or similar approaches

– Identify variations

Cap

ture

tes

ting

idea

s


Testing Ideas

• Creative approaches – Action verbs

– Mind Maps

– Soap Operas

– Lateral Thinking

Cap

ture

tes

ting

idea

s


State Models

power up

idleinserting

coins

userchoose

makecoffee

serviceneeded

coin insertedreset button

coin return

right amountenteredcoin return

button pushed

no cupsOR no coffeeOR sensor jam

cup removed

Testing IdeasC

aptu

re t

estin

g id

eas


Testing Ideas• Data

– Flow– Structure– Create– Update– Change

Cap

ture

tes

ting

idea

s


Testing Ideas

• Environment– Hardware

– Software

– Operating systems

– Locales

– Browsers

– Plug-ins

– Co-dependent software

Cap

ture

tes

ting

idea

s


Testing Ideas

• White Box– Design

– Internal structure

– Code

Cap

ture

tes

ting

idea

s


Testing Ideas• Bug taxonomies

– Collections of possible bugs– Appendix A of Testing

Computer Software, Kaner, Falk, Nguyen

– Boris Biezer Taxonomy Otto Vinter manages

– Shopping cart taxonomy Giri Vijayaraghavan

Cap

ture

tes

ting

idea

s



Which test?

•Impact estimation–For each test idea guesstimate:

• benefit of implementation

• consequence of implementation

• benefit for not implementing

• consequence of not implementing

–How credible is the information?

Tria

ge te

stin

g id

eas


Which test?

Test Idea Rejection – What If?–If the cost/benefit does not make business sense then consider implementing:• part of the test, could that lead to part of the

benefit at a more reasonable cost?

• more than the stated test, would that generate more benefit?

• a different test than the stated idea, could that generate more benefit for less cost?

Tria

ge te

stin

g id

eas


Test Triage

• Test Triage Meeting– Review Context

• Business

• Technical

• Organizational

– New Information• Test results

• Bug results

• New testing ideas

Tria

ge te

stin

g id

eas


Test Triage

• Allocate Testing Assignments to Testers– Make sure testers know context

– Best thing to test

– Best person to test it

– Best people to explore it

– Best lead

– Are subject matter experts required

Tria

ge te

stin

g id

eas


Test TriageLife of a test idea

a. Comes into existence

b. Clarified

c. Prioritizeda. Test Now (before further testing)

b. Test before shipping

c. Nice to have

d. May be of interest in some future release

e. Not of interest in current form

f. Will never be of interest

d. Integrate into a testing objective

Tria

ge te

stin

g id

eas


Which test is next?

• Questions– Given state of project, state of business, state of

technology, our abilities, our experience and our history, what we know and what we do not know, what should we test next?

– How much effort are we willing to spend continuing to test this project?

– Can we ship yet?

Tria

ge te

stin

g id

eas


Deciding what not to test?

• Time pressure

– Should we skip a test?

– If test failed could system still be of value to some stakeholder?

– If test was skipped could important bugs have been otherwise found?

Tria

ge te

stin

g id

eas


Guidelines and Decisions

• To each stakeholder– risk of failure

– consequence of failure

– value of success

– how much certainty do we have

– is it a wild guess or an absolute truth?

Get

Sta

rted

Rig

ht


Bottom Line

• My experience is that it is better to omit a test on purpose than to skip it because you ran out of time or forgot about it!

Get

Sta

rted

Rig

ht • Systematically collecting, evaluating and triaging testing ideas helps me decide what not to test - at least for now?


Thank You

• Questions?

just-in-time testing workshop - let's...

Documents