not your grandfathers requirements-based testing webinar – robin goldsmith, philip lew
TRANSCRIPT
XBOSoft, Inc. All Rights Reserved. 1
Not Your Grandfather’s
Requirements-Based Testing:
Do Agile User Stories or ATDD Fix It?
with Robin Goldsmith
#TestCaseTips
XBOSoftDedicated to Software Quality Improvement
Founded in 2006
We speed products to market with our expert:
• Software QA consulting
• Software testing
Global team with offices in San Francisco & Beijing
“Thorough, accurate and fast”
XBOSoft, Inc. All Rights Reserved. 2
House Rules
Participants other than the speakers are muted
Ask questions in the GoToWebinar control on the
right side of your screen or through Twitter
@XBOSoft
Questions may be asked throughout the webinar -
we’ll try to answer them at the end
You’ll receive info on recording after the webinar
Webinar Hashtag: #TestCaseTips
XBOSoft, Inc. All Rights Reserved. 3
Meet Our Speakers
Philip Lew
CEO and Founder, XBOSoft
• Relevant specialties and
passions
o Software quality process,
evaluation, measurement
and improvement
o Software quality in use /
UX design
o Mobile User Experience
and usability
o Cycling and travel
XBOSoft, Inc. All Rights Reserved. 4
Rachel Geibig
Marketing Manager, XBOSoft
• Emails you about our
industry hot topic
webinars
• Invites you to download
our latest trends in
software testing white
papers
XBOSoft Team
Robin Goldsmith
XBOSoft, Inc. All Rights Reserved. 5
Author of:
• ArtechHouse book Discovering REAL
Business Requirements for Software
Project Success
• Forthcoming book Cut Creep—Put
Business Back in Business Analysis to
Discover REAL Business Requirements for
Agile, ATDD, and Other Projects
• Numerous articles and featured
presentations
• The Proactive Testing™, Proactive SQA™,
REAL ROI™, and Beyond the Textbook
Software Acquisition™ methodologies
• Subject expert on requirements and testing for TechTarget’s
SearchSoftwareQuality.com, and subject expert/reviewer for the IIBA Business
Analysis Body of Knowledge (BABOK v2)
• Works directly with and trains business/systems professionals in quality and testing,
requirements analysis, software acquisition, project management and leadership,
metrics, ROI, and process improvement
www.gopromanagement.com
http://psqtconference.com/
Keynote and Featured Presentations
by World Renowned Quality Leaders and Experts
Ed AdamsCEO, Security
Innova
Jennifer BonineVP of global
delivery and
solutions for
tap|QA
Isabel EvansIndependent
Consultant
Philip LewCEO, XBOSoft
Robin GoldsmithInternational
Institute for
Software Testing
XBOSoft, Inc. All Rights Reserved. 7
Bhushan Gupta,
Gupta Consulting, LLC.
Robin Goldsmith,
Go Pro Management, Inc.
Johanna Rothman,
Rothman Consulting Group, Inc.
Peter Khoury,
Magnetic Speaking
Rex Black,
RBCS
Darlene Greene,
Consultant
PNSQC 2016 Keynote and Invited Speakers www.PNS.org
1. Take your
requirements
2. Test that each
one has been
satisfied
correctly
3. Duh!
“In grandfather’s day, requirements-based tests were no big deal.”
Why We’re Here
XBOSoft, Inc. All Rights Reserved. 9
Requirements-based tests
• Are fundamental
• Seem obvious
Except
1. What requirements?
2. What are requirements?
3. How to identify tests based on requirements?
Objectives
You should be able to:
• Recognize various test design techniques that
some equate with requirements-based testing
• Describe the significance of testing against REAL
business requirements vs. against
product/system/software requirements
• Critically evaluate effectiveness of tests based on
user stories and their acceptance criteria
Is one of your issues testing that requirements are met?
Awareness is a start.
XBOSoft, Inc. All Rights Reserved. 11
Agenda
XBOSoft, Inc. All Rights Reserved. 12
• Low-level test design techniques that some
claim = requirements-based testing
• REAL business requirements distinctions and
significance for testing
• Challenges identifying the requirements
• Strengths and issues of Agile user stories and
their acceptance tests
What Does “Requirements-Based
Testing” Mean to You?
Check all that apply1. Confirming documented requirements have been met
2. Confirming undocumented requirements are met
3. Relying on users to do the testing
4. Using decision tables, decision trees, and/or cause-
and-effect graphing to identify business rule
conditions
5. Exercising use cases
6. Using some other specific test case design technique
7. Other
Pitfall:Gurus Equate RBT with Particular
(Low-Level) Techniques
• Appropriating the term, typically implying it’s only one or a few specific structured methods to design test cases by systematically elaborating each business rule, e.g.
• Decision trees and tables
• Logic mapping (including Cause and Effect Graphing), like white box but of functionality
XBOSoft, Inc. All Rights Reserved. 14
Pitfall (continued)
• Exercising use cases, albeit systematically
• Focusing only on such methods can be forest-and-trees
• Can mask other, especially bigger, missed requirements
• Large number of tests identified can be overwhelming
• Yet, still can overlook significant requirements to test
XBOSoft, Inc. All Rights Reserved. 15
Some Equate RBT to
Decision Trees
Find by ID Order
Exit
Find by Phone No.
Search by Name
Search Name in Zip
Add Vendor
More Equate to Decision Tables
Especially helpful for business rules.
Is this correct?
Conditions
Actions
Test Cases
Scenarios
ID match Y N
Phone match Y N
Name select Y N
NameZip select Y N
Add Vendor Y
Order Y Y Y Y Y
Decision Tables Issues
• Still easy to miss conditions
• Does not reflect sequence, or
• Helps with combinations of conditions, to a point
Conditions
Actions
Test Cases
Scenarios
ID match Y N N N N N
Phone match Y N Y N N N N N N
Name select Y N Y Y N N N
NameZip select Y N Y Y Y N
Add Vendor Y Y Y
Exit Y
Order Y Y Y Y Y Y Y Y Y Y Y Y Y
Exit Y
Degrees of Structural Coverage
1. Execute module/routine from entry to exit
Within a module/routine, execute all:
2. Lines of code/steps
3. Branches (basis paths, complete)
4. Logic paths (exhaustive)
• Flowgraph: Node
Module’s Entry, Exit
Where logic branches
Junctions (logic returns)
• Flowgraph: Edge
all logic between nodes
IF
IF
T F
TF
“Flowgraph”
Others Equate to Cause-Effect
Graphing
Paid
Grace
At
fault
UninsuredPay_
Claim
or
and
~ not
Pay the damage claim when
the insured is at fault or the
accident is caused by an
uninsured motorist and their
premium has been paid, or
when they are within the
payment grace period.Tool uses structural paths approach to
optimize rigorous tests of all variants of
multiple sequenced decisions
Still More Equate to Executing
Use Case Paths/Scenarios
Defined as “how an actor interacts with the system.”
The actor is usually the user, and the system is what the
developers expect to be programmed. Therefore, use cases
really are white box/design rather than black box/business
requirements. Flowgraph this Use Case. Path=Test Case
U1.Enter Vendor ID R1.1 Vendor is found (U4)
R1.2 Vendor is not found (U2)
U2.Enter vendor name R2.1 Select vendor from list (U4)
R2.2 Vendor is not in list (U3)
U3.Add vendor R3 Vendor is added (U4)
U4.Enter order R4 Order is entered (Exit)
Flowgraph of Use Case
U1
R2.2
Exit
U3
R3
R1.2R1.1
U4
R2.1
U2
R4
• U1-R1.1-U4-R4-Exit
• U1-R1.2-U2-R2.1-U4-R4-Exit
• U1-R1.2-U2-R2.2-U3-R3-U4-
R4-Exit
• U1-R1.2-U3-R3-U4-R4-Exit
• U1-R1.2-Exit
• U1-R1.2-U2-R2.2-Exit
• U1-R1.2-U2-R2.2-U3-Exit
• U3-R3.1-Exit
• U4-R4.1-ExitR3.1
R4.1
Equating to Any Specific (Typically
Business Rule-Oriented) Technique
• Can’t be right—excludes requirements and/or tests
identified by other claimed techniques
• Does not address other types of requirements
– Functionality
– Calculations, formulas, algorithms
– Data variants and database structure
– Quality factor “-ilities,” e.g., usability, security,
performance, reliability, maintainability, etc.
• Irrelevant for overlooked requirements
XBOSoft, Inc. All Rights Reserved. 23
Pitfall: RBTs Are Only as Good as the
Requirements
• Most organizations’ aren’t very good – unclear, wrong, incomplete – and are actually even worse than realized because most miss REAL requirements
• The REAL requirements aren’t what you happened to write down, they’re what you ought to have known about and captured so they aren’t forgotten, including things like quality factors and performance levels.
• Most of what people call “requirements” actually are design. Design doesn’t have to be engineering or technical. A product or system one intends to create is a design, albeit high-level.
Error Sources by Phase
Portion of
System Correct
Portion of
System In ErrorPhase
Requirement missed,
extra, or misdefined
Requirements
Required and defined appropriately
Included in design and designed
appropriately
Design
Included in product/system and
built appropriately
Build & Test
Included in finished system
and installed appropriately
Implementation
Missing from design
or inappropriate
Missing or built
inappropriately
Missing from or
inappropriate in finished system
IT (and Other?) Project Economics
• Maintenance is 66-90% of system cost
• Maintenance is mainly completion/ correction of
development (wrong/missed requirements)
• 2/3 of finished system errors are requirements
and design errors
• Fixing a requirements error will cost
• 10X+ during development/construction
• 75-1000X+ after installation (maintenance)
Do your organization’s routine measures show these effects?
What’s the relevance to
requirements-based tests?
XBOSoft, Inc. All Rights Reserved. 27
Two Types of Requirements:Business/User/Customer Product/System/Software
• Business/user/stakeholder/ customer language & view, conceptual; exist within the business environment
• Serves business objectives
• What business results must be delivered to solve a business need (problem, opportunity, or challenge) and provide value when delivered/satisfied/met
• Language & view of a human-defined product/system
• One of the possible ways: How
(design) presumably to accomplish the presumed business requirements
• Often phrased in terms of features/external functions each piece of the product/system must perform to work as designed (Non-/Functional Specifications)
Many possible ways to accomplish
Even Requirements “Experts” Think
the Difference Is Just Level of Detail
Business Requirements
(High-Level, Vague)
Product/
System/
Software
Reqs.
(Detailed)
BABOK v3 2.3 p. 26
“Business requirements: statements
of goals, objectives, and outcomes
that describe why a change has been
initiated.”
Mistaken “Levels Model”
When Business/User Requirements
Are Detailed First, Creep Is Reduced
Business Requirements(High-Level)
Business
Product/System/Software
Reqs.(High-Level)
Reqs.
(Detailed)Reqs.
(Detailed)
Product/
System/
Software
Requirements Overview
StakeholdersBusiness needs, problems, value
Discovery
Analysis
High-Level & DetailedREAL Business/ Stakeholder Requirements Deliverable Whats Value
Product/System/ Software Requirements Features Hows
Respond to
Functional RequirementsUse CasesSoftware Requirements Specifications
[Non-Functional Requirements]Quality Factors, Attributes, ‘Ilities’(Supplemental Specifications)
User/
(Usage)
High-LevelDetailedTechnical/EngineeringDesign
Code
Tests Based on Product/System/
Software Features/Requirements
• What’s usually meant by “requirements”
• Necessary but not sufficient
• Can pass even though product is not right
XBOSoft, Inc. All Rights Reserved. 32
“Conventional We’s Dumb”
More Effective Tests Are Based on
REAL Business Requirements
• Demonstrate that the
• Product/system/software how’s
• Satisfy the REAL business requirements
deliverable what’s
• And thereby provide REAL value by
achieving objectives
XBOSoft, Inc. All Rights Reserved. 33
“Unconventional Wisdom”
Pitfall: Just User Story Requirements
and Their Acceptance Test
• Assuming all that’s needed is writing user stories and acceptance tests for them
• User story brevity means lots of requirements content may be taken for granted or simply overlooked
• Ditto for user story acceptance criteria/tests, easily not as accurate or thorough as assumed
• Several common Agile dynamics can cause user story requirements content to deteriorate
• Consider effects of discarding
• And…
Requirements in Agile Are
Generally Considered User Stories
As a <type of user>
I <want/can/am able to/need to/etc.>
so that <some reason>
Mike Cohn
“User Stories, Epics and Themes”http://www.mountaingoatsoftware.com/blog/stories-epics-and-themes
Some call “placeholder for conversation”
User Stories Theory
StakeholdersBusiness needs, problems, value
Discovery
Analysis
High-Level & DetailedREAL Business/ Stakeholder Requirements Deliverable Whats Value
Product/System/ Software Requirements Features Hows
Respond to
Functional RequirementsUse CasesSoftware Requirements Specifications
[Non-Functional Requirements]Quality Factors, Attributes, ‘-ilities’(Supplemental Specifications)
User/
(Usage)
High-LevelDetailedTechnical/EngineeringDesign
Code
UserStories
CONVERSATIONS
User Stories Actually Are a Bit
More
• Card
• As a <role>
• I want <something>
• So that <benefit>
• Conversation
• Confirmation
• User story acceptance criteria, tests
What do conversations do to the user
stories requirements and their tests?
Working code
Too Often User Stories Practice
StakeholdersBusiness needs, problems, value
Discovery
Analysis
High-Level & DetailedREAL Business/ Stakeholder Requirements Deliverable Whats Value
Product/System/ Software Requirements Features Hows
Respond to
Functional RequirementsUse CasesSoftware Requirements Specifications
[Non-Functional Requirements]Quality Factors, Attributes, ‘Ilities’(Supplemental Specifications)
User/
(Usage)
High-LevelDetailedTechnical/EngineeringDesign
Code
UserStories
Conver
sation
REAL Business Requirement:
Calculate 5% Sales Tax on the Order
• Requirements-Based Test:
• Enter one item at $99.99, Sales tax should be $5.00
• Acceptance Criteria
• Create an order consisting of both taxable and
nontaxable items
• Create an order that covers more than one page
• Produce a credit voucher for a returned item
• Calculate for states and localities with different rates
Content is the important part, not format.
= or reveal requirements?
You Should Be Able to:
• Recognize various test design techniques that
some equate with requirements-based testing
• Describe the significance of testing against
REAL business requirements vs. against
product/system/software requirements
• Critically evaluate effectiveness of tests based
on user stories and their acceptance criteria
Go Pro Management, Inc. Seminars/Consulting--Relation to Life Cycle
Proactive Systems/Software Quality Assurance (SQA)™
Software, Test Process Measurement & Improvement
Feasibility
AnalysisSystems
AnalysisSystem
DesignDevelop-
ment Implement-
ation Operations
Maintenance
Proactive Testing™:
Risk-Based Test Planning,
Design, and ManagementTesting Early in the Life Cycle
Re-Engineering: Opportunities for IS
Credibly Managing Projects and Processes with Metrics
21 Ways to Review Requirements
Making You a Leader
Managing Software Acquisition and Outsourcing:
> Purchasing Software and Services> Controlling an Existing Vendor’s Performance
Proactive User Acceptance TestingReusable Test Designs
Test EstimationRisk
Analysis
Defining and Managing
Business Requirements
Writing Testable SW Requirements
Post your webinar questions on Twitter @XBOSoft
Registrants will receive an email with information on where to view the recording
and slides from today’s webinar.
Join us to keep updated on all our webinars, reports and white papers:
facebook.com/xbosoft
+xbosoft
linkedin.com/company/xbosoft
Check out our blog: http://xbosoft.com/software-quality-blog/
Download our free white papers: http://xbosoft.com/knowledge-center/
Email us with ideas for future webinars or questions regarding our services!
Thank you!
Q+A
www.xbosoft.com