planning your agile testing: a practical guide

1/22/2013

1

Janet Gregory, DragonFire Inc.Copyright 2013

Star Canada 2013

With material from Lisa Crispin

Copyright 2013 DragonFire Inc.

My experience comes …As a tester, working on agile teams

Coaching and training, learning

Agile Testing: A Practical Guide for Testers and Agile Teams; Addison Wesley 2009

1/22/2013

2


How many years experience with agile?◦ At least 2 years◦ Less than 2 years◦ Less than 1 year◦ Have done some reading, starting in a team◦ Completely new to the concept

Are you a ...◦ Tester◦ Test Manager◦ Programmer◦ Other

3


10 minutes ~1 minute / person

NameWhere you are from?Why you are here?

Let everyone have a chance to introduce themselves

4

1/22/2013

3

Copyright 2013 DragonFire Inc. 5


• have short iterations• encourage active customer participation• demand whole team collaboration• test features & stories as they are coded• deliver business value at regular intervals• adapt their processes based on feedback

6

1/22/2013

4

Sequential, phased – eg. waterfall

7

It 1 It 2 It 3

A B A BC

Time

Agile: iterative, incremental– Each story is expanded, coded and

tested– Possible release after each iteration

Requirements

SpecificationsCode

TestingRelease

It 0

A BCD

It 4

A BCD

FE


What concerns you about agile

and test planning?

1/22/2013

5


Test Approach – The Agile WayProject Initiation Get an understanding of the project (business)

Participate in sizing stories Create Test Plan Release/Project Planning

Each Iteration1 ….X

Estimate tasks, Run regression testsCollaborate with customers on acceptance tests

Write, automate and execute new story testsPair test with other testers, developers

Perform exploratory testing

Perform Final Load TestComplete Final Regression Test

Perform UATPerform Mock Deploy

Participate in Release Readiness

System Test / End Game

Participate in Release to ProdParticipate in Retrospectives

Release to Prod/ Support

9


Story Tests

Project Test Plan

10

Release Test Plan

1/22/2013

6


• Product Roadmap◦ High level feature ideas

• Release Planning◦ Backlog of sized and prioritized stories

• Iteration Planning◦ Backlog of estimated tasks◦ Estimations are in hours: how long will it take

11


• Types of testing needed• Automation• Complexity• Size

More details later …

12

1/22/2013

7


Instead of saying NO, or being the gatekeeper

Be the information provider so business can make the decisions

Questions… on agile approach to planning

13


1/22/2013

8


Let’s talk a little testing ….

What is enough to keep you out

of trouble?


• Find bugs• Check correctness• Test reliability• Check usability• Answer “Is it done?”• Learn about the application• Feedback into future stories

What else???

16

1/22/2013

9


• Each group, collaborate • Write different types of testing that you do• One type per sticky• 5 minutes

17


• Model for classifying tests• Looks at the purpose of the tests – the ‘why’

18

1/22/2013

10

19

Agile Testing Quadrants Brain child of Brian Marick


• Take your tests and put them in the quadrant you think they belong in.

• Were you able to get them all classified?

• Add tests as you think about them.

20

1/22/2013

11

Agile Testing Quadrants (Brian Marick)

21


• Can be used as a communication toolo to explain testing in a common language

• Emphasize whole-team responsibilityo focus on collaborationo whole team participation

• Help plan what tests to automateo and tools and infrastructure needed

1/22/2013

12


For each story / feature◦ No story is done until tested◦ Customer needs captured as passing tests◦ Automated regression tests

For release readiness◦ Delivers value◦ “Doneness” in all quadrants

23


1/22/2013

13


Unit Testso Tests developer intent - program designo Tests a small piece of code o Makes sure it does what it shouldo TDD

Component Testso Tests architecture intent – system designo Tests that components work together correctly

Connectivity Tests


• Builds quality in• Focus on internal code quality• Builds testability into code• Go faster, do more

o Unit tests provide refactoring support• Provides instant feedback • TDD increases confidence in design• Forms the foundation of automation suite

1/22/2013

14

27

Can you think

of anything else

for Quadrant 1?


1/22/2013

15


• Use to elicit requirements• Acceptance Test (or Example) Driven Development

o Allows developers to code until the tests pass• Capture examples, express as executable tests• User experience

o wire frames; mock-ups / prototypes


• Help customers achieve advance clarity• Gain shared common understanding of stories• Drive development with business-facing tests• Obtain enough requirements to start coding• Focus is external quality• Customer – developer – tester collaboration◦ Power of Three

1/22/2013

16

31

Can you think

of anything else

for Quadrant 3?

Discussions?


1/22/2013

17


• Exploratory testingo what didn’t we think about

• Test for usabilityo understand end users, personas

• Tours• User Acceptance Testing (UAT)


• Iteration reviews / demoso Builds confidenceo Quick feedback loop

• Pair test with customers• Informal demos

o Pair exploratory testing with customer

1/22/2013

18


• Provide feedback ….o Turn learnings into tests that drive new featureso Change process as needed

• Evaluation of the product• Recreate actual user experiences• Realistic use


• Know your customers• Make them real• Plan your exploratory

testing using them

• Picture – from Jeff Patton’s Pragmatic Personas weekly column on Sticky Minds (1/25/2010)

1/22/2013

19


• Think of some personas or characters and devise exploratory scenarios which that role might get into, for example:

Shopping website - Amazono Senior citizen who never shopped on the webo Hacker looking to cause troubleo Working mom in a rusho Internet-savvy teen

37 37

38

Any questions about

Quadrant 3?

1/22/2013

20



• Non-functional tests• Performance, scalability, stress, load• “ility” testing• Automated deployments• Memory management• Infrastructure testing• Security testing

o Roles & permissions , system ‘hacking’• Data migration• Recovery

1/22/2013

21

• May be higher priority than functional req’ts• Makes the ‘finished’ product• Transfer specialized expertise


Consider the four quadrants • Would you add anything?• Consider your quality attributes

• Do you need to move any tests?• What tests are you missing?• What is your team is not doing now?

42

1/22/2013

22

push the tests lower

Automate at the feature

level

Automate at the story

level

Automate at the task

level

Sequential, phased – eg. waterfall

44

It 1 It 2 It 3

A B A BC

Time

Agile: iterative, incremental– Each story is expanded, coded and

tested– Possible release after each iteration

Requirements

SpecificationsCode

TestingRelease

It 0

A BCD

It 4

A BCD

FE

All automation done here

Automation hereAutomation

here

Automation here

Automation here

1/22/2013

23




Questions… on types of testing?

45


1/22/2013

24


Release Level Test Planning


• Consider scope, priorities, risks• Document only what is useful• Consider all types of testing• Budget time for:

o infrastructureo toolso automation needs

• Strive for simplicity

48

1/22/2013

25

49

Copyright 2013 DragonFire Inc.50

• Sizing of stories◦ think about impacts to system

• Ask questions ◦ that may impact the ‘bigness’ of the story

◦ to uncover hidden assumptions

◦ such as ‘what if?’, or “what happens if?”

• Not the time for details• Create a project “test plan”

1/22/2013

26


• Are we working with a vendor?◦ If so, how do we coordinate testing?

• Are there dependencies between … ◦ features?◦ teams?◦ Stories?

51

1/22/2013

27


• Should be project specific• High-light critical information◦ risks◦ assumptions◦ constraints

• Focus on what is really needed• Put static information –in a Quality

Management Strategy document

53


1/22/2013

28



A test matrix (one example)◦ provides a view of the release◦ can also be used as visible progress

But, remember◦ the value is in the planning

Let`s look at an example

56

1/22/2013

29


Copyright 2009 Janet Gregory, DragonFire 57


Let’s do some collaboration,

& work together to find a solution

1/22/2013

30


Sub topic

Sub topic

Sub topic

MAINTOPIC

Sub topic

Sub topic

first time

change

new account

passwordrules

encryption

save user name

rules

59


Mind map testing ideas for a feature

How many ideas can you generate in 5 minutes?

1/22/2013

31


Feature A• As an on-line shopper, I want to provide my

shipping address when I check out so that my order goes to the right location

Feature B• As an on-line shopper, I want to provide

payment information when I check out so I am billed correctly.


• Create a test matrix for the stories on the next slide.

• Functionality down the left side• Test conditions across the top

o Think heuristics o Use mnemonics (ex. SFDPOT)

• Gray out the squares that are not applicable.

1/22/2013

32


Feature A• As an on-line shopper, I want to provide my

shipping address when I check out so that my order goes to the right location

Feature B• As an on-line shopper, I want to provide

payment information when I check out so I am billed correctly.


• What value do you see in something like this?• What did you learn?• How would you use it?

1/22/2013

33




Questions on test planning at the release level?

65


1/22/2013

34


Iteration Planning: • Stories tell us about the tip of the iceberg• What don’t we know?• What questions should we ask to find out?

68

1/22/2013

35


Create a user story

Write Customer (Q2) Tests

Start thinking

how to code

Expand tests –

Story TestsAutomate Q2 Tests

Exploratorytesting

TDD

Customer User

Acceptance

Product ownerProduct owner/ TesterTesterTester/ProgrammerProgrammer

Pair,“Show Me”

Copyright 2013 DragonFire Inc.70

• Be proactive – preplanning • Try specifications workshops – Gojko Adzic

• Help customer achieve ‘advance clarity’ on stories◦ Customers speak with “one voice”◦ Testable stories◦ Steel threads◦ Create acceptance tests

1/22/2013

36


• Define high level story tests or examples • Find hidden assumptions• Define and estimate testing tasks – consider:

o automation needso test data, o exploratory testingo Q4 tests (‘ilities’, security, performance, etc.)


Ask questions• What's the business goal?• Can the user mess up?• What’s the best thing?• What’s the worst thing that can happen?• Watch for scope creep or ‘bling’• Is this story testable?

72

1/22/2013

37


Picture from Mike Cohn’s website

73

To Review


• Express the intent of the story• Use examples as specific instances of a scenario• Think expected and unexpected behaviors• Gives a shared common understanding of the

story• Feeds into TDD (Test Driven Development)

74

1/22/2013

38


As a new user, I want to create an account with a user name and

password so that only I can access my information.

===============================

75 75


• To create my account, I enter a valid user name and password, my information is saved and I am logged into the system and on the home page.

• If I enter an invalid user name, I get an error message “Invalid User Name”, and I am able to try again.

• If I enter an invalid password, I get an error message “Invalid Password”, and I am able to try again.

76

1/22/2013

39


BDD – Behavioural Driven Development

Given the user has no existing accountWhen she requests to create a new account,Then she enters a valid user name and valid

password (rules defined) And the information is saved upon submitting .

77 77


returnValue TestLogin ( userName, password)

User Name Password Expected result comments

JanetGregory Password Access system as JanetGregory

Valid combo saved

Janet Gregory Password Error Space in user name

JanetGregory Abc Error Invalid password

1/22/2013

40


• Write Acceptance Test(s) for one of the two stories – any format you like

• To help, try mind-mapping or draw a flow diagram

• Ask your customer for examples.• Think “Amazon”

• (~15 min)


Story a – Part of Feature A• As the company shipper, I need to verify the

city, state and postal code so that the order goes to the right location.

Story b – Part of Feature B• As the company accountant, I need to ensure

the credit card information is correct so the on-line shopper is billed correctly.

1/22/2013

41

We walk out of the iteration planning meeting ....

Now what?


1/22/2013

42


• Start simple with the high level acceptance tests◦ Add boundary, edge conditions, etc.◦ Add non-functional tests

• Experiment to find the right detail level• Review with programmers◦ Then automate

• Keep adding until story is complete• Consider exploratory test scenarios

83


Acceptance Tests – Fit Automation Style-- Third thread

84

User Name Password Expected result Comments

JanetGregory Password Access system as JanetGregory

Valid combo saved

Janet Gregory Password Error Space in user name

Janet#Gregory Password Error Special char not allowed

Password Error Blank user name

JanetGregory Password Error User already exists

JanetGregory Abc Error Not enough char in password

JanetGregory Error Blank password

1/22/2013

43


How do you know you are done?

85


Description Completed

1 Story tests reviewed, automated and pass Yes

2 Exploratory testing complete Yes

3 Unit tests reviewed and pass Yes

4 Q4 tests complete Yes

5 Acceptance tests pass Yes

86

1/22/2013

44




Questions on test planning at the story level?

87


1/22/2013

45


• Understand your context• Understand the purpose• Consider ROI (return on investment)• Push the tests lower• Automate the repetitive and boring tests• Plan, but document simply• Plan, but plan for the appropriate level


• How much is too much?• What is not enough?• Why do you need them?• What is the right information?• Who looks at them?• What do they use them for?• What is the simplest way you can capture

them?

90

1/22/2013

46


• What problem are you trying to solve?• What measurements make sense?• What is the simplest way to gather them?• Who needs to see them?• What is the simplest way to make them visible?• How are you going to use them?• When will you know to stop?• Be Aware!!!

o The wrong measurements can be deadlyo Metrics can be misinterpreted


• How are you going to know when you are done?• What is enough?• Collaborate to decide what coverage you want• Think risk – include the customer• Functional coverage• Lines of code? … be careful• What tools can you use?

o Heuristicso Mnemonics

1/22/2013

47


• Understand the level of precision you need◦ Think big picture in release planning◦ Think tasks during iteration planning◦ Think tests during story test planning

• Use ATDD for feature and story testing• Make it visible• Make it simple• Make it valuable• You want to be testing, not documenting




Are there still unanswered questions.. Concerns?

94

1/22/2013

48


Now AvailableAgile Testing: A Practical Guide for Testers and Agile Teams

By Lisa Crispin and Janet Gregory

www.agiletester.ca

My contact infowww.janetgregory.cahttp://janetgregory.blogspot.com/Email: [email protected]://lisacrispin.comEmail: [email protected]

95


• www.lisacrispin.com • [email protected]• http://www.exampler.com - Brian Marick’s web site• www.testobsessed.com (heuristics cheat sheet) – Elisabeth Hendrickson• http://www.shino.de/blog/ - Markus Gärtner’s blog• Gojko Adzic, Bridging the Communication Gap, 2009• Gojko Adzic, Executable Specifications, Manning, 2011• vwww.mountaingoatsoftware.com – Mike Cohn’s web site (and all his

books)• http://www.developsense.com/2009/04/of-testing-tours-and-

dashboards.html • James Whittaker, Exploratory Software Testing, 2009 Addison-Wesley• http://www.stickyminds.com/ Pragmatic Personas - Jeff Patton’s weekly

column 1/25/2010• Jean Tabaka, Collaboration Explained, 2006 Addison-Wesley• Agile Manifesto: http://agilemanifesto.org/

96

planning your agile testing: a practical guide

Technology

acceptance tests

ounit tests

planning copyright

test planning

new story tests pair

test approach

requirements acceptance

agile approach