specification-by-example with gherkin
DESCRIPTION
Many teams struggle with the implementation of user story acceptance criteria and having a shared understanding about the expected story outcomes. This often results in missed stakeholder expectations, ad-hoc assumptions made by the team during implementation and conflict between team members and the product owner around testing. This session shows how specification-by-example and acceptance test driven development will address team conflict, missed stakeholder expectations and overall increasing the level of clarity on the project end-to-end. The presentation will cover the theory behind ATDD and case-studies and practical experience from real projects. The talk was held at the ALM summit 3 in Redmond, January 2013. Recording of the talk can be found here: http://channel9.msdn.com/Events/ALM-Summit/ALM-Summit-3/Implementing-ATDD-and-Specification-By-ExampleTRANSCRIPT
Almost 4 years later …
#106 in Visual Studio Gallery (30.1.2013)
#135 in NuGet (30.1.2013)
~800 visits daily
~ 4000 active users > 30 contributors
Goals
Make you curious about Specification-By-Example
New perspective for SpecFlow users
About me • Managing Partner at TechTalk • Agile consulting and delivery • Based in Europe:
Vienna, Budapest, Zurich
Terminology
• Specification-By-Example
• Acceptance Test Driven Development (ATDD)
• Behaviour Driven Development (BDD)
Agile requirements
Purpose of user stories?
• Describe user needs
• Unit of prioritization
• Unit of planning
• Reminder for a discussion
• Mechanism to defer detail
Requirement levels Impact Mapping
Story Mapping
Specification-By-Example
Why?
How?
Acceptance Criterion
Epic
Deliverable, Outcome
Impact, Goal
Easier to define upfront Harder to define upfront
Reminder for a discussion
Bug report
Isolated, formalized example
User Activity
User Story
Example
Code
Collecting acceptance criteria
“I would try to put a book into the shopping cart …”
“I would try to remove a book from the shopping cart…”
“I’d check whether the shopping cart is empty, when I enter the shop …”
“I would try to add the same book again to the shopping cart …”
Books can be placed into shopping cart.
Books can be removed from shopping cart.
Shopping cart should be empty when entering the shop.
Adding same book again to shopping cart should increase quantity.
As a potential customer I want to collect books in a shopping cart So that I can order several books at once.
“Imagine this story is already implemented: What would you try out?”
Using examples
Examples for user stories
UI wire frames, existing UI
rules, key examples
existing artifacts, mock-ups
Discussion of acceptance criteria
Original idea for the illustration: George Dinwiddie http://blog.gdinwidiee.com
public void TestInitialOrderDiscount()
{
Customer newCustomer = new Customer();
Order newOrder = new Order(newCustomer);
newOrder.AddBook(
Catalog.Find(“ISBN-0955683610”)
);
Assert.Equals(33.75,
newOrder.Subtotal);
}
Register as “bart_bookworm” Go to “/catalog/search” Enter “ISBN-0955683610” Click “Search” Click “Add to Cart” Click “View Cart” Verify “Subtotal” is “$33.75”
We would like to encourage new users to buy in our shop. Therefore we offer 10% discount for their first order.
Specification-By-Example Examples …
– make abstract descriptions better understandable
– are usually not formally exchanged or documented
Brian Marick
Examples Tests
Requirements
consist of
describe verify fulfillment of
Discussion of acceptance criteria
Original idea for the illustration: George Dinwiddie http://blog.gdinwidiee.com
public void TestInitialOrderDiscount()
{
Customer newCustomer = new Customer();
Order newOrder = new Order(newCustomer);
newOrder.AddBook(
Catalog.Find(“ISBN-0955683610”)
);
Assert.Equals(33.75,
newOrder.Subtotal);
}
Register as “bart_bookworm” Go to “/catalog/search” Enter “ISBN-0955683610” Click “Search” Click “Add to Cart” Click “View Cart” Verify “Subtotal” is “$33.75”
We would like to encourage new users to buy in our shop. Therefore we offer 10% discount for their first order.
Illustrated with shared examples
Original idea for the illustration: George Dinwiddie http://blog.gdinwidiee.com
Given the user has not ordered yet When the user adds a book with the price of USD 37.50 into the shopping cart Then the shopping cart sub-total should be USD 33.75.
Discover new aspects
Original idea for the illustration: George Dinwiddie http://blog.gdinwidiee.com
Actually, this not quite right: Books on sale should be excluded.
Collaboration: 3 amigos “Happy Path”
Technical feasability
Exceptions, border cases
Abstract acceptance criteria As a shop visitor I want to collect books in my shopping basket so that I can purchase multiple books at once.
Books can be added to the shopping basket
Books can be removed from the shopping basket
Shopping basket is initially empty
The same book can be added multiple times to the shopping basket
Formalizing examples As a shop visitor I want to collect books in my shopping basket so that I can purchase multiple books at once.
Books can be added to the shopping basket
Given my shopping basket is empty
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 1 copy of “Harry Potter”
Formalizing examples As a shop visitor I want to collect books in my shopping basket so that I can purchase multiple books at once.
Books can be added to the shopping basket The same book can be added multiple times to the shopping basket
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
Given my shopping basket contains 1 copy of “Harry Potter”
Example structure
The same book can be added multiple times to the shopping basket
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
Given my shopping basket contains 1 copy of “Harry Potter”
Title: Describes isolated, focused intention=abstract acceptance criterion
Arrange: Context, initial state of the system
Act: Execution of the feature Assert: Assertion of observable behavior
And I should see the warning: “Book already existed in basket”
Triple-A constraint “Checks”
Steps can be chained up
Evolution of a domain specific test and specification language.
Life time of examples
Purpose of examples
• Shared understanding: acceptance criteria
• Documentation: system details
• Regression-tests: violated assumptions
Continuous validation
When I add the book “Harry Potter” to my shopping basket
Then my shopping basket should contain 2 copies of “Harry Potter”
Given my shopping basket contains 1 copy of “Harry Potter”
„Step Definitions“ are binding individual steps to an automatable interface of the application.
Automation does not necessarily have to bind to the UI.
Automation of system is supported/evolving with development.
System UI Automation
Demo
Gherkin automation for .NET
• Visual Studio plugin (VS-Gallery)
• NuGet Package
http://www.specflow.org
Impact on testing
Test automation becomes expensive, when …
• trying to automate manual tests
• making tests unreadable when automating them
• automating only after completing implementation
structure
readability
when to test
Structure Manual tests
Asserts Multiple combined features
Structure ACT-ASSERT- ACT-ASSERT- ACT-ASSERT- …
Dependent on existence of other features Long test path with high chance to break Cause and impact of an error hard to trace.
Automated Check
Single aspect of a single feature
ARRANGE – ACT – ASSERT
Independent of other features Short test path with lower chance to break Cause and impact of an error easier to trace.
Test automation pyramid
User journeys
Acceptance-criteria
Units many
harder
easier
Au
tom
ati
on
few
exploratory testing
Source: Mike Cohn
Readability // Go to web page 'http://localhost:40001/' using new browser instance BrowserWindow localhostBrowser = BrowserWindow.Launch( new System.Uri(this.RecordedMethod1Params.Url)); // Click 'Register found item' link Mouse.Click(uIRegisterFoundItemHyperlink, new Point(56, 9)); // Click 'Save' button Mouse.Click(uISaveButton, new Point(44, 14)); int foundItemNo1 = int.Parse(uIFundNr127Pane.InnerText.Substring(9)); // Click 'Register found item' link Mouse.Click(uIRegisterFoundItemHyperlink, new Point(63, 7)); // Click 'Save' button Mouse.Click(uISaveButton, new Point(34, 11)); int foundItemNo2 = int.Parse(uIFundNr128Pane.InnerText.Substring(9)); Assert.IsTrue(foundItemNo1 + 1 == foundItemNo2); // Click 'Close' button Mouse.Click(uICloseButton, new Point(26, 11));
A readable test case Scenario: New found items should receive a consecutive number for the current year Given the previous found item of the current year had the number 145 When I register a new found item Then the last found item of the current year should have the number 146
When to test
Acceptance criteria (ATDD, BDD)
Unit Tests (TDD)
business view
technical view
Exploratory tests Workflow tests
Performance, Scalability, Usability,Security, …
de
fin
ing
th
e p
rod
uct
criticizing
the
pro
du
ct
New dimension: defining the product Synergy: Specification of requirements and tests
Agile Testing Quadrants: Brian Marick
Manual testing is still necessary!
User journeys
Acceptance-criteria
Units
exploratory testing
Manual Check after story done
Main success paths
Undiscovered acceptance criteria
No/(few) manual regression checks
Few paths are enough
More time for exploration
Cross-functional work
Sprint 1 Sprint 2 Sprint 3
short iteration
US4 Plan
Implement & autom. test
US5 Plan
Implement & autom. test
US2 Plan
Implement & autom. test
US3
Plan
Implement & autom. test
US6 Plan
Implement & autom. test
US1 Plan
Implement & autom. test
US7 Plan
Implement & autom. test
US8 Plan
Implement & autom. test
US9 Plan
Implement & autom. test
Explo
ratory Tests
Specification and test
Collaboration for defining acceptance criteria
Collaboration for automation
Preventing bugs instead of finding them!
Extension of “Test Cases”
Limit WIP
Collaboration in manual
testing
Impact on development
TDD workflow
Extending TDD for business
Transparency for stakeholders
In Progress
Current sprint report: all scenarios
Starting with first scenario
Finishing first scenario
Progressing on scenarios
Progressing on scenarios
Progressing on scenarios
Progressing on scenarios
First user story ready for testing
Earlier start of manual testing
Done work can break again
Done work can break again
Business readable error reports
Living documentation
Higher level scenario structure
Story Mapping
Why?
How?
Acceptance Criterion
Epic
Deliverable, Outcome
Impact, Goal
User Activity
User Story
Example
Code Easier to define upfront Harder to define upfront
Story Maps
• Original concept by Jeff Patton
• Support iterative product design
• Optimized for desired outcome or deliverable the system should support
Building story maps Get popular books delivered fast and
conveniently
Find book
Collect books
Order Wait for delivery
Receive delivery
time
browse best
sellers
shopping basket
enter address
delivery slip
delivery notification
pay with credit card
search book by
title wish list
inquiry order status
desired outcome or deliverable
user activities
system features
nec
essi
ty
Walking skeleton
Enabling build–measure-learn Get popular books delivered fast and
conveniently
Find book
Collect books
Order Wait for delivery
Receive delivery
time
browse best
sellers
enter address
delivery slip
pay with credit card
search book by
title wish list
inquiry order status
nec
essi
ty
shopping basket
delivery notification
manual work-
around
omitted steps
desired outcome or deliverable
user activities
system features
Story Map example
Sprint 1
Sprint 2
Sprint 3
Sprint 4
Dropped Scope
Added scope
User Stories vs. Features Product/Sprint Backlog
User Story 1 AccCrit 1
AccCrit 2
User Story 2 AccCrit 3
AccCrit 4
Living Documentation
Feature 1
AccCrit 1
AccCrit 2
Feature n AccCrit 4
AccCrit m User Story n
AccCrit 5
AccCrit m
AccCrit 3 AccCrit 5
„Done“
• Future options of the system • Organized/refined according to
priority, value, effort, risk, ... • Next possible increments of
the product (units of work)
• Current state of the system • Organized/refined for
functional overview • Versioned and maintained
together with source code
Living documentation • Link business
readable automation from source code to story maps
• Feed execution results into story maps
• Sync Story Story Map requirements with TFS work items
Lessons learned
Collaboration over specification
Documentation after conversation • Collaborative discovery • Should trigger new questions • Infinite number of examples • Readability and ability to automate
Level of automation
Controller
Business Layer
Data Layer
Model
View
Browser automation
Trigger behavior through controller
Assert behavior on model, db, ..
Setup pre-conditions through service interfaces
Out-of-process
In-process
Test execution performance
• Grouping tests – Current WIP
– Completed work
• Database – In-memory
– Templates for setup
• Parallel execution
• Smart execution order
• Level of automation
Internal vs. external DSL
Example Source: Liz Keogh https://github.com/lunivore/tictactoe-java/blob/master/ scenarios/com/lunivore/tictactoe/scenarios/Three_in_a_row_wins.java
Non-functional acceptance criteria
Given there are 100,000 users registered on the system
When I create a new account
Then I should be taken to my dashboard within 5ms
When 1000 users are hitting the homepage simultaneously Then each user should get a response within 2ms
Matt Wynne http://blog.mattwynne.net/2012/03/13/using-cucumber-for-load-testing
Tools
Gherkin automation tools
www.cukes.info
www.behat.org
www.specflow.org
Ruby, Java, JavaScript, C++
.NET, Mono, Silverlight, WP7
PHP
Books
Bridiging the Communication Gap Gojko Adzic
Specification by Example Gojko Adzic
Explore It! Elisabeth Hendrickson
Exploratory Software Testing James Whittaker
Summary
Specification-by-example
• Collaborative discovery and specification
• Impact on development and test
• Combine with other agile reqmt methods
• Can be supported by tools
Questions
Got curious about Specification-By-Example?
Learned something new about using SpecFlow?
Christian Hassa, TechTalk ([email protected], @chr99ha)
Join the BoF Session „Specification-By-Example (ATDD, BDD) in practice“ Today at 15h15 (right after this talk)