(exploratory) testing with scripted support

(Exploratory) TestingWith Scripted Support

Simon Morley October 2014

Experience & Considerations

Saturday 25 October 14

My Situation Context• Complex Environment

• Meaning overhead & set-up costs

• Meaning preparation, tooling or ordering

• Many signaling protocols

• Meaning script support for signal injection and check

• Agile & Incremental SW Development


Testing in Agile

• A common “agile” development model:• Backlog item / Opportunity Analysis

• Plan

• Design

• Test

• Build

• Iterate


Testing in Agile

• A common “agile” development model:• Backlog item / Opportunity Analysis

• Plan

• Design

• Test

• Build

• Iterate

Where testing prep & execution happens


Testing before “testing”• Testing in the “Opportunity Analysis”,

“Planning” & “Design” “phases” might be:

• Feature Investigation - from a testing perspective

• Testable?

• Creates issues with existing framework, tools and maintenance?

• Feature Walkthrough


Possible Testing Approach• Investigate the new feature

development

• Develop framework and tools, develop scripts, comms, test scripts & framework & feature, comms, debug scripts, comms, debug environment, comms, test feature, comms, report completion, archiving.


Feature Investigation

• Feature Walkthrough

• Test environment impacts & walkthrough

• Test Analysis summary


Test Analysis Summary• Describe* or reference the current

feature, test environment and framework situation

• Describe* impacts to test environment and framework

• Describe* approach

• Include coverage goals for product, feature and testing


Comms?• Communication

• Can be: Verbal & Written

• Can be: Formal & Informal

• Always: Two-way!

• Reporting - stand-ups, progress boards, meetings

• Reflect & decide about next steps


Reflect & Decide?• What has the testing so far revealed?

• Product coverage goals met?

• Testing coverage goals met?

• Perceived risk acceptable?

• Stakeholder view on these?


Reflect & Decide #2

• Next steps?

• Additional test steps?

• Enough already?

• Change direction or approach?


Reflect & Decide #3• Pause & Reflect

• Control Q: “What don’t we know?”

• Silent Evidence

• Note: This pause & reflection is important - see thinking about heuristic discovery.


Necessary support?• Can the tools & framework support

changes?

• Is the tool & framework architecture modular?

• Remember to test the tool architecture!!!

• Do the tools & framework allow quick prototyping?

• If not -> fix it!


Describe*• Describe: Meaning the level sufficient in

the team or project.

• If coordination is needed between teams this should be a format that is commonly understood (picture/symbols, dashboard or text document).

• You agree what’s appropriate!


Exploratory Testing Traps• Sometimes people think of ET means

less:-

• Documentation

• Structure

• Traceability

• And more

• Random banging on a keyboard


Exploratory Testing Traps• Sometimes people think of ET means

less:-

• Documentation

• Structure

• Traceability

• And more

• Random banging on a keyboard

Wrong !!!


ET Traps #2• Actually ET is about more

responsibility in skilled testing

• The skilled tester is responsible for communicating his/her approach, thinking, results and any mandated archiving


ET Traps #2• Actually ET is about more

responsibility in skilled testing

• The skilled tester is responsible for communicating his/her approach, thinking, results and any mandated archiving I.E. Plenty of structure,

traceability & documentation

where appropriate!


ET Traps #3• No support for skilled testers and

testing.

• Support for skilled testing is a pre-requisite (hand-in-hand) to realizing emphasized tester responsibility.


ET Traps #3• No support for skilled testers and

testing.

• Support for skilled testing is a pre-requisite (hand-in-hand) to realizing emphasized tester responsibility.

Fix these if you want good & valuable testing!


Discussion• Exploratory testing can be performed

with scripting support

• Good testing (in complex environments) requires elements of:

• Domain knowledge - built through early investigation

• Test environment knowledge & understanding


Discussion #2

• Plus

• Rapid learning

• System and problem analysis

• Decision making

• Framing, Consensus & Commumnication


References• Experience Report: Introducing Exploratory

Testing

• Deliberated Exploration - A Walk in the Woods

• Experiences with Semi-Scripted Exploratory Testing

• Framing: Some Decision Analysis and Frames in Testing

• Silent Evidence in Testing

• On Thinking about Heuristic Discovery


http://www.ministryoftesting.com/2010/02/experience-report-introducing-exploratory-testing/




http://testers-headache.blogspot.se/2010/11/deliberated-exploration-walk-in-woods.html

http://testers-headache.blogspot.se/2010/11/deliberated-exploration-walk-in-woods.html

http://www.slideshare.net/YorkyAbroad/experiences-with-semiscripted-exploratory-testing




http://testers-headache.blogspot.se/2011/08/framing-some-decision-and-analysis.html




http://testers-headache.blogspot.se/2012/03/silent-evidence-in-testing.html

http://testers-headache.blogspot.se/2012/03/silent-evidence-in-testing.html

http://testers-headache.blogspot.se/2014/04/on-thinking-about-heuristic-discovery.html

http://testers-headache.blogspot.se/2014/04/on-thinking-about-heuristic-discovery.html

Thanks!• Simon Morley

• Twitter: @YorkyAbroad

• Blog: testers-headache.blogspot.com

• Questions?

• Get in touch!


Feature Walkthrough


(exploratory) testing with scripted support

Technology