specification-by-example with gherkin

FROM STAKEHOLDER EXAMPLES TO LIVING DOCUMENTATION

Christian Hassa - ch@techtalk.at - @chr99ha .NET User Group Bern, 5th of December 2012

Specification-By-Example with Gherkin

TechTalk at a glance

• Agile consulting and delivery • Offices in: Vienna, Budapest, Zurich • Established in 1993

Vienna/Austria

3

How do we specify

requirements?

4

The purpose of user stories …

•Describe user needs •Describe system features •Unit of planning • Reminder for a discussion •Mechanism to defer a discussion

5

Impact Mapping

Story Mapping

Specification-By-Example

Establishing a shared understanding

Why?

How?

Code

Acceptance Criterion

Epic

Capability

Impact, Goal

Easier to define upfront Harder to define upfront

Reminder for a discussion

Bug report

Isolated, formalized example

User Activity

User Story

Example

6

Collecting Acceptance Criteria

“I would try to put a book into the shopping cart …”

“I would try to remove a book from the shopping cart…”

“I’d check whether the shopping cart is empty, when I enter the shop …”

“I would try to add the same book again to the shopping cart …”

Books can be placed into shopping cart.

Books can be removed from shopping cart.

Shopping cart should be empty when entering the shop.

Adding same book again to shopping cart should increase quantity.

As a potential customer I want to collect books in a shopping cart So that I can order several books at once.

“Imagine this story is already implemented:

how would you verify it?”

7

Why do we need

examples?

8

UI wire frames, existing UI

rules, key examples

existing artifacts, samples

Examples for user stories

10

Specification-by-Example

Examples … • make abstract descriptions

better understandable

However … • examples are usually not formally

exchanged or documented

Brian Marick

Examples Tests

Requirements

consist of

describe verify fulfillment of

11

Discussion of acceptance criteria

public void TestInitialOrderDiscount()

{

Customer newCustomer = new Customer();

Order newOrder = new Order(newCustomer);

newOrder.AddBook(

Catalog.Find(“ISBN-0955683610”)

);

Assert.Equals(33.75,

newOrder.Subtotal);

}

Register as “bart_bookworm” Go to “/catalog/search” Enter “ISBN-0955683610” Click “Search” Click “Add to Cart” Click “View Cart” Verify “Subtotal” is “$33.75”

We would like to encourage new users to buy in our shop. Therefore we offer 10% discount for their first order.

Original idea for the illustration: George Dinwiddie http://blog.gdinwidiee.com

12

… illustrated with formalized examples

Given the user has not ordered yet

When the user adds a book with the price of EUR 37.5

into the shopping cart

Then the shopping cart sub-total is EUR 33.75.

Original idea for the illustration: George Dinwiddie http://blog.gdinwidiee.com

13

Discover hidden assumptions

Actually, this not quite right: Books on sale should be excluded.

Original idea for the illustration: George Dinwiddie http://blog.gdinwidiee.com

14

Collaboration: 3 amigos

“Happy Path”

Technical feasability

Exceptions, border cases

Original idea for the illustration: George Dinwiddie http://blog.gdinwidiee.com

15

Abstract acceptance criteria

As a shop visitor I want to collect books in my shopping basket so that I can purchase multiple books at once.

Books can be added to the shopping basket

Books can be removed from the shopping basket

Shopping basket is initially empty

The same book can be added multiple times to the shopping basket

16

Examples for acceptance criteria

As a shop visitor I want to collect books in my shopping basket so that I can purchase multiple books at once.

Books can be added to the shopping basket

Given my shopping basket is empty

When I add the book “Harry Potter” to my shopping basket

Then my shopping basket should contain 1 copy of “Harry Potter”

17

As a shop visitor I want to collect books in my shopping basket so that I can purchase multiple books at once.

Books can be added to the shopping basket

Examples for acceptance criteria

When I add the book “Harry Potter” to my shopping basket

Then my shopping basket should contain 2 copies of “Harry Potter”

The same book can be added multiple times to the shopping basket

Given my shopping basket contains 1 copy of “Harry Potter”

18

The same book can be added multiple times to the shopping basket

Structure of examples

Given my shopping basket contains 1 copy of “Harry Potter”

When I add the book “Harry Potter” to my shopping basket

Then my shopping basket should contain 2 copies of “Harry Potter”

Title: Describes intention/abstract acceptance criterion

Arrange: Context, initial state of the system

Act: Execution of the feature

Assert: Assertion of observable behaviour

And I should see the warning: “Book already existed in basket”

Triple-A constraint “Checks”

Chaining up steps

20

How long do we

care about the

examples?

21

Purpose of the examples

• Shared understanding: acceptance criteria

• Documentation: Look-up detail aspects of the system

• Regression-tests: Understand what assumptions have been violated

22

Continuous validation with automation

Given my shopping basket contains 1 copy of “Harry Potter”

When I add the book “Harry Potter” to my shopping basket

Then my shopping basket should contain 2 copies of “Harry Potter”

System

„Step Definitions“ are binding individual steps to an automatable interface of the application.

Automatable interface

UI Automation

Automation does not necessarily have to bind to the UI.

Automatability of system is supported/evolving with development.

23

Demo

http://www.specflow.org

Gherkin automation for .NET • Visual Studio plugin (VS-Gallery) •NuGet Package

24

What does that mean for

testing?

25

UI automation becomes expensive when …

• trying to automate

manual tests

• making tests

unreadable when

automating them

• automating after

completing

development

structure

readability

point in time

26

Structure Manual tests

Asserts Multiple combined features

Structure ACT-ASSERT- ACT-ASSERT- ACT-ASSERT- …

Dependent on other features Long test path with high chance to break

Automated Check

Single aspect of a single feature

ARRANGE – ACT – ASSERT

Independent of other features Short test path with lower chance to break

27

Test automation pyramid

User journeys

Acceptance-criteria

Units

exploratory testing

Source: Mike Cohn

many

few

hard

easy

Au

tom

ata

bili

ty

28

// Go to web page 'http://localhost:40001/' using new browser instance BrowserWindow localhostBrowser = BrowserWindow.Launch( new System.Uri(this.RecordedMethod1Params.Url)); // Click 'Register found item' link Mouse.Click(uIFundstückerfassenHyperlink, new Point(56, 9)); // Click 'Save' button Mouse.Click(uISpeichernButton, new Point(44, 14)); int fundNr1 = int.Parse(uIFundNr127Pane.InnerText.Substring(9)); // Click 'Register found item' link Mouse.Click(uIFundstückerfassenHyperlink, new Point(63, 7)); // Click 'Save' button Mouse.Click(uISpeichernButton, new Point(34, 11)); int fundNr2 = int.Parse(uIFundNr128Pane.InnerText.Substring(9)); Assert.IsTrue(fundNr1 + 1 == fundNr2); // Click 'Close' button Mouse.Click(uICloseButton, new Point(26, 11));

Readability

29

A readable test case Scenario: New found items should receive a consecutive number for the current year Given the previous found item of the current year had the number 145 When I register a new found item Then the last found item of the current year should have the number 146

30

When to test (point in time)

Acceptance criteria (ATDD, BDD)

Unit Tests (TDD)

business view

technical view

Exploratory tests Workflow tests

Performance, Scalability, Usability,Security, …

def

inin

g th

e p

rod

uct

criticizing th

e p

rod

uct

New dimension: defining the product Synergy: Specification of requirements and tests

Agile Testing Quadrants: Brian Marick

31

Cross-functional teams

Co-creation Fast feedback

32

Cross-functional work

Sprint 1 Sprint 2 Sprint 3

short iteration

US4

Plan

Implement & autom. test

US5

Plan

Implement & autom. test

US2

Plan

Implement & autom. test

US3

Plan

Implement & autom. test

US6

Plan

Implement & autom. test

US1

Plan

Implement & autom. test

US7

Plan

Implement & autom. test

US8

Plan

Implement & autom. test

US9

Plan

Implement & autom. test

Explo

ratory Tests

Specification and test

Collaboration for defining acceptance criteria

Collaboration for automation

Preventing bugs instead of finding them!

Extension of “Test Cases”

Limit WIP

Collaboration in manual

testing

33

Manual Testing is always necessary!

User journeys

Acceptance-criteria

Units

exploratory testing

Source: Mike Cohn

many

few

harder

easier

Au

tom

ata

bili

ty

Manual Check after Story Done

Main success pathes

Undiscovered acceptance criteria

No/(few) manual regression checks

Few pathes are enough

More time for exploration

34

What does that mean for

development?

35

Workflow - TDD

36

Extending TDD for business

37

Transparency for stakeholders

In Progress

38

Current sprint report: all sprint scenarios

39

Starting with first scenario (AC)

40

Finishing the first scenario (AC)

41

Progressing scenario after scenario

42

Progressing scenario after scenario

43

Progressing scenario after scenario

44

Implementing user stories in parallel

45

First user story ready for testing

46

Manual testing can start even earlier

47

Already done work can break again

48

Already done work can break again

49

See what is temporarily not working

51

Living documentation

52

Story Mapping

Optimizing and refining scope

Why?

How?

Code

Acceptance Criterion

Epic

Capability

Impact, Goal

Easier to define upfront Harder to define upfront

User Activity

User Story

Example

53

Story Maps

• Original concept by Jeff Patton • Supports iterative product design • Optimized for desired outcome or

deliverable the system should support

56

Building story maps

Get mainstream book fast and conveniently

delivered

Find book I want

Collect books

Complete order

Wait for book

Receive book

time

browse best

sellers

put into basket

enter address

receive delivery

slip

receive delivery notificat.

pay with credit card

search book by

title

create wish list

inquiry order status

desired outcome or deliverable

user activities

system features

ne

cess

ity

57

Walking skeleton

Enabling build – measure - learn

Find book I want

Collect books

Complete order

Wait for book

Receive book

time

browse best

sellers

enter address

receive delivery

slip

pay with credit card

search book by

title

create wish list

inquiry order status

put into basket

receive delivery notificat.

ne

cess

ity

manual workaround

omitted steps

Get mainstream book fast and conveniently

delivered

60

Story Maps in ALM with TFS

TFS work items in story maps

http://speclog.net

61

Story Map in SpecLog

62

Sprint 1

63

Sprint 2

64

Sprint 3

65

Sprint 4

66

Dropped Scope

67

Added Scope

69

User Stories vs. Features

Product/Sprint Backlog

User Story 1 AccCrit 1

AccCrit 2

User Story 2 AccCrit 3

AccCrit 4

Living Documentation

Feature 1

AccCrit 1

AccCrit 2

Feature n AccCrit 4

AccCrit m User Story n

AccCrit 5

AccCrit m

AccCrit 3 AccCrit 5

„Done“

• Future options of the system • Organized/refined according to

priority, value, effort, risk, ... • Next possible increments of

the product (units of work)

• Current state of the system • Organized/refined for

functional overview • Versioned and maintained

together with source code

71

Living documentation

• Link business readable automation from source code to story maps • Feed execution results into story maps

72

Additional

thoughts

73

Level of automation

Controller

Business Layer

Data Layer

Model

View

Browser automation

Trigger behaviour through controller

Assert behaviour on model, db, ..

Setup pre-conditions through service interfaces

Out-of-process

In-process

74

Performance

• Grouping tests

• Current WIP

• Completed work • Database

• In-memory

• Templates for setup • Parallel execution • Smart execution order

75

Internal vs. external DSL

Example Source: Liz Keogh https://github.com/lunivore/tictactoe-java/blob/master/ scenarios/com/lunivore/tictactoe/scenarios/ Three_in_a_row_wins.java

76

Non-functional acceptance criteria

Given there are 100,000 users registered on the system When I create a new account Then I should be taken to my dashboard within 5ms

Given 1000 users are hitting the homepage simultaneously Then each user should get a response within 2ms

Matt Wynne http://blog.mattwynne.net/2012/03/13/using-cucumber-for-load-testing

77

Tools

78

Gherkin automation tools

www.cukes.info

www.behat.org

Ruby, Java, JavaScript, C++

www.specflow.org

.NET, Mono, Silverlight, WP7

PHP

79

80

Books

Gojko Adzic Bridiging the Communication Gap

Gojko Adzic Specification by Example

Elisabeth Hendrickson Explore It!

81 Christian Hassa: ch@techtalk.at - @chr99ha