Experiences with Semi-scripted Exploratory Testing

Simon Morley

Agile Testing Days 201115 Nov 2011

About

Me I work as a Test Coordinator with many

different teams and testers. I jump in as a team leader from time to time And even do some testing!

Context Multimedia IP networks To “telecom grade”

Overview

Preamble... Case story

Background / Problem Description

Approaches

Observations & Lessons

Preamble

My usage of...

Scripted Exploratory Semi-Scripted

Semi-Scripted & Exploratory?

What?

Experiences with Combining the following approaches

Scripted Testing Scripted Testing vs Scripted Execution

Exploratory Testing

Scripted Execution vs Scripted Testing?

Another Example: Login to PC

Step1: Enter User-name Step2: Enter Password

What if password has expired? What if it was a guest account (no password)?

Result: Successfully logged-in

Semi-Scripted?

“Walking in the woods” Using some pre-defined set-up as an enabler to an

exploratory session

The pre-defined set-up should not exclude observation and feedback.

Walking with someone else is valuable

Why “Walking in the woods” Seasons change → set-up/config changes

Repeating walks can actually help reveal new information (due to conditions changing)

Terrain changes → systems “change” due to use

Prolonged usage of systems build up different problems/conditions “under the surface”

Case Story

Background Feature Walkthroughs Test Ideas & Feasibility Test Execution Some Success Indicators Challenges and Lessons

Background

Experience Context Background factors – challenging... Short deadline Trial Feature

Complex Environment

Traditional Approach

Environment

NENE(SUT)

SimNE

NE

NE NE

SimNE

SimNE

SimNE

Unchanged

Changed

Result?

Additional terms of reference

Catch-up effort Gather information for feature assessment

“Free hand”

Possibilities...

Approach Possibilities

Initial Feature Analysis

Feature Walkthrough

Test Feasibility

Learning

Feature Walkthrough All were 'up to speed' before the discussion

All the team plus external experience

Discussion – Q&A - Brainstorming

Whiteboard is centre-stage

Good toolsmiths make a difference!

Feature Walkthrough → Brainstorming for risks

Brainstorming #1

Brainstorming #2

First real testing of the feature!

Test Ideas

Test Ideas → Test Feasibility

Learnings

Gather the right peopleto look at the problem.

Allow time for reflection.

Estimation Notes

Main estimation figures were:

Who was needed and for how long

Equipment need & availability (feedback)

Estimation was not based on test case ideas

Test Execution

Semi-Scripted?

Sameinput

Sameinput

ConfigChanges

ProvisionedDataChanges

NENE(SUT)

SimNE

NE

NE NE

SimNE

SimNE

SimNE

Test Execution

Test Execution

Test Execution

Test Execution

Test Execution

No “test idea constraints”

linked with tester thinking & reflection

from execution

feeds back into better test ideas.

When you're not interested in numbers

Then testers are no longer interested in talking about

them!

Results from execution

Found issues not found elsewhere

Results used to declare better information on the feature when going to trial

Start of change in mindset

Some Success Indicators

A Success Indicator...

A test report changes from

“Run X test cases, where Y failed and Z passed”

And becomes more like..

“Feature A tested, there are some interaction issues with feature B and, by implication, risks for features C and D.

We have not explicitly tested feature C.

Recommend some further investigation with feature A in combination with features B and C.

As feature D is not to be used in trial a priority decision is needed on the inclusion of D into scope.

If feature D is to be put into general usage then further test and investigation is recommended.”

Pilot activity spread to end-2-end teams with

greater engagement and end-value appreciation.

Issues in the product for end-user usage & interactions are found → adding value to the product reporting

Observation & Analysis

What's happening here?

Testing has been framed as an investigative activity!

Scripting is re-framed as a toolrather than a goal for testing!

Note on labels The testing was never labelled as one form or another

The frames in which it was presented/discussed changed

Questions to trigger the activity changed

Move away from valuing test cases

Move to relate the value in PO terms

Sometimes labels get in the way!

Framing a choice differently → transition!

Change? How?

This is a transition rather than a big bang change. Why? Adaptive, small steps, fail fast & improve!

Time was key – pull the team in a direction, but allow them to see that the benefit from the change → then they control the speed of change and so it happens quicker!

The team is challenged to think about the activity – and with the right support & encouragement they generally do think more!

Factors particular to this case

“Free hand”

No expectation on change of mindset

First pilot was in a separated test team.

To some external observers...

Maybe not so much different

A product with some level of testing is delivered

But, from the team perspective

The value of the testing and ideas was debated & discussed early

All in the team & PO value this earlier feedback

Ideas formed, tried out, re-evaluated

Challenges Getting past test case counting

Relating value in test ideas to end-user value

Working with good test story telling is important

Whole team involved in initial analysis

Prototype early – good toolsmiths are important

Recap Big Challenges

Used the challenges as a means to start the transition

Part of transition was enabled by using scripting as an enabler

But also to trigger thinking / questioning mindset → throughout (workshop->test analysis/feasibility->execution)

Lessons

Lessons #1

Feature Walkthrough: First real testing of the feature happens early Gather the right people to look at the problem. Allow time for reflection.

Test Ideas: No “test idea constraints”. Scripts are an enabler not the goal.

Lessons #2

Execution No “test idea constraints” linked with tester

thinking & reflection from execution feeds back into better test ideas.

When you're not interested in numbers then, then testers are no longer interested in talking about them!

Lessons #3 General

Challenge the team (testers) to think and they generally will think!

Allow the team to see the benefits of change (transition) – then they will begin to drive the speed of change.

Frame testing differently Frame scripts as tools and enablers rather than

fountains of knowledge!

Lesson #4 It takes practice to get your testing experience from:

To:

Questions?

Blog: http://testers-headache.blogspot.com

Twitter: @YorkyAbroad

Thank You!

Attributions http://www.flickr.com/photos/kalavinka/

http://www.flickr.com/photos/kalavinka/4617897952/

http://www.flickr.com/photos/tesla314/2666463779/

http://www.flickr.com/photos/txmx-2/6054712319/

http://www.flickr.com/photos/28385889@N07/3058090229/in/photostream/

http://www.flickr.com/photos/somegeekintn/3810233454/

http://www.flickr.com/photos/sashomasho/260136080/