Download - Top Challenges in Testing Requirements
T13
Requirements
5/8/2014 1:30:00 PM
Top Challenges in Testing
Requirements
Presented by:
Lloyd Roden
Lloyd Roden Consultancy
Brought to you by:
340 Corporate Way, Suite 300, Orange Park, FL 32073
888-268-8770 ∙ 904-278-0524 ∙ [email protected] ∙ www.sqe.com
Lloyd Roden
Lloyd Roden Consultancy
With more than twenty-eight years in the software industry, Lloyd Roden has worked as a developer, test analyst, and test manager for many different organizations. Lloyd was a consultant/partner with Grove Consultants for twelve years. In 2011 he created Lloyd Roden Consultancy, an independent UK-based training and consultancy company specializing in software testing. Lloyd’s passion is to enthuse, excite, and inspire people in the area of software testing. He has spoken at conferences worldwide including STAREAST, STARWEST, Better Software, EuroSTAR, AsiaSTAR, and Special Interest Groups in software testing in several countries. In 2004, he won the European Testing Excellence award.
Top challenges in testing requirements Keynote Presentation
Written by Lloyd Roden www.lloydrodenconsultancy.com Version 1_0 © Lloyd Roden
© Lloyd Roden Testing Requirements 131014
testing requirements-1
Top Challenges in Testing Requirements
1
Contents
Introduction Top challenges for testing requirements
Rising to the challenge
2
© Lloyd Roden Testing Requirements 131014
testing requirements-2
What is a requirement?
Definition: ! something that is required/needed ! something essential to the existence of something else
why do we have requirements?
to make money
requirement for a new product
to improve
requirement for an existing system
to remain relevant
requirement for changing technology
3
What do requirements look like?
specifications
use cases
user stories
story boards
emails verbal
some forms are better than others 4
© Lloyd Roden Testing Requirements 131014
testing requirements-3
Root Cause Analysis shows…
56%
27%
10% 7%
Defects Originate in Project Phase
Requirements Design Code Other
similar statistics from other projects • 58% requirement bugs for large mobile phone company • 73% requirement bugs for large financial company
Source: Binder & Associates
5
Problems that we have with requirements
" non-existent requirements " expectations are not fully understood
" evolutionary requirements " constantly changing the scope of the project
" wrong or bad requirements " wasted time in development and testing
" vague or ambiguous requirements " leading to misunderstandings
" too much detail in the requirements " lack of creativity
" too complex requirements " unable to test effectively
" too many requirements " overbearing and frustrating 6
© Lloyd Roden Testing Requirements 131014
testing requirements-4
Contents
Introduction Top challenges for testing requirements
Rising to the challenge
7
Challenge #1: No requirements
" can we test with no requirements? " we do it all the time
" performance, usability…
" should we test with no requirements? " challenge is knowing whether something is right
“I know it when I see it”
“I want something, but I don’t know
what”
“and by the way, could you give me
an estimate”
8
© Lloyd Roden Testing Requirements 131014
testing requirements-5
requirement
Challenge #2: Evolutionary requirements
requirement idea
design
code
test
9
The good and bad of evolutionary requirements
" excellent when customers don’t really know what they want
" often produces systems that meet customer’s expectations
" high collaboration
" documentation can be sparse/non-existent
" tester’s nightmare " automation usually not
possible until final iteration
10
© Lloyd Roden Testing Requirements 131014
testing requirements-6
Challenge #3: Bad or wrong requirements
some bad products…
• all started out as “bad” ideas/requirements • time “wasted” testing them
we need to recognise and challenge any bad/wrong requirements 11
Challenge #4: Vague/ambiguous requirements
which do you consider to be “good” requirements " the help messages must be meaningful and informative " the fields provided must all have context sensitive help " the system must be user-friendly " the system must be reliable " navigation must be consistent across all applications " the system updates must be fast " the website must sustain 5000 simultaneous users " the mobile phone application must be secure " developers must be able to fix a bug within 2 days
why are some requirements better than others?
12
© Lloyd Roden Testing Requirements 131014
testing requirements-7
What is wrong with these requirements?
" requirement 1
" requirement 2
online orders can be made at anytime, but will be processed on the next working day. Payment must be made using PayPal or standard credit/debit cards. Orders will be dispatched within 2 days. Customers will be notified via email if a partial order has been sent and can return their order within 28 days and obtain a full refund.
working day standard within
partial within full
A computer program plays chess with one user at a time. It displays the board and pieces on the screen. Moves are made by dragging the pieces
13
Challenge #5: Too much detail in the requirement
" is there such a thing of having too much detail in requirements?
" why are too much detail in requirements a challenge? " it often prevents creativity in development and test " mistakes are made due to a lack of questioning " too much detail could lead to conflicting
requirements
14
© Lloyd Roden Testing Requirements 131014
testing requirements-8
Challenge #6: Complex requirements
and we call this progress…
1990 2013
1990 2013
1990 2013
“yo, I am outside
your door. lol
15
Challenge complexity at every opportunity
" simplicity seen as weak and uninteresting " who wants a “basic mobile phone?”
" complex is seen as good " I don’t understand this,
so it must be really good (everyone else understands)
" $1m pen
suggestion: challenge requirements and design documents at every opportunity to see whether
complexity is needed 16
© Lloyd Roden Testing Requirements 131014
testing requirements-9
Features and functions used
Jim Johnson XP2002 Standish Study Group
Features and Functions Used16%
13%
7%
19%
45%Sometimes Used
Often Used
Always Used
Rarely Used
Never Used
Features and Functions Used
20%
64%
16%
Often and Alw aysUsed
Rarely or Never Used
Sometimes
this means we have driven up complexity by putting in things
that are not required
17
”.. If there is too little demand on them, people are bored. If there is too much for them to handle, they get anxious. Flow occurs in that delicate zone between boredom and anxiety.”
Perf
orm
ance
Arousal level
Flow
Strained concentration
Boredom
Distracted
Csikszentmihalyi
Challenge #7: Too many requirements
low medium high
low
high
m
ediu
m
Stress
Laid back
Anxious
Panic/Anger/Fear
Optimum Performance
18
© Lloyd Roden Testing Requirements 131014
testing requirements-10
MoSCoW
" Must " describes a requirement that must be satisfied in the final
solution for the solution to be considered a success
" Should " represents a high-priority requirement that should be in the
solution if possible. It is often a critical requirement but one which can be satisfied in other ways if necessary
" Could " describes a requirement which is considered desirable but
not necessary. This will be included if time and resources permit
" Wont " represents a requirement that stakeholders have agreed will
not be implemented this time but may be considered for the future 19
Contents
Introduction Top challenges for testing requirements
Rising to the challenge
20
© Lloyd Roden Testing Requirements 131014
testing requirements-11
Choose the most appropriate lifecycle
Integration Testing
Acceptance Testing
System Testing
Component Testing Code
Design Specification
System Specification
Business Requirements
sequential iterative agile
• detailed requirements • complex requirements • vague requirements
• no requirements • evolutionary requirements • vague requirements
• too many requirements • complex requirements • vague requirements
there is no lifecycle applicable for bad/wrong requirements 21
" an agile development approach to component testing
Use Test-Driven Development
pick a req.
yes no
need more tests for requirement?
write enough code
to pass the test
write component test
automate test using a tool
22
© Lloyd Roden Testing Requirements 131014
testing requirements-12
Use test design techniques as a review technique
" example requirement:
by reading this requirement we may have a few questions, or we might think we understand what is required. However…
A thermostat displays the temperature of a greenhouse from 15oC to 35oC. If the temperature falls below 20oC then “too cold”
appears on the display and a blue light flashes. It the temperature exceeds 30oC then “too hot” appears and a red light
flashes.
23
Review using test case design techniques
" sharpens our understanding " generates tests " finds more defects with requirements Tests for our example: Temperature Expected result 17oC, 15oC, Too Cold, blue light flash 20oC, 25oC, 30oC No display, no light? 32oC, 35oC Too hot: red light flash 10oC, 40oC ???
2 lights or 1 light?
whole degrees?
up to and including…
where is the temperature taken…isolated or whole room? 24
© Lloyd Roden Testing Requirements 131014
testing requirements-13
Produce some requirement guidelines
do not use pronouns do not use unqualified adjectives and adverbs do not use the word “etc” is the requirement is testable?
do we have the necessary expertise? are the requirements prioritised?
does the requirement need breaking down?
is the requirement clear and understood?
25
Use walkthroughs to clarify and understand
" walkthroughs are a powerful static testing technique " to educate, aid understanding and find bugs in documents
" the power and full potential of the walkthrough is often not recognized or utilized " watch for what is said and how it is said " do not underestimate body language " resist peer pressure and challenge when
you don’t understand
26
© Lloyd Roden Testing Requirements 131014
testing requirements-14
Use the Quality Attribute Template
Concept Test Scale Plan Must Now Best completeness compare
menu and help items
percentage of menu
items in the help
100% 90% 40% 100%
ease of access
sample 25 features
average number of keystrokes
to find a feature
3 7 12 2
accuracy
collect data each
month
number of reported defects in the help
<5 20 50? 0
Usa
bilit
y O
n-lin
e he
lp
high level
attribute
sub attribute
descriptio
n
of attri
bute how
is it to be
measured what
is being
measured measure
to be
achieved worse
measure
acceptable curre
nt
measure ultim
ate
measure
particularly useful for vague non-functional requirements 27
Perform Exploratory Testing
" it is not obvious what the next test should be
" we want to go beyond the obvious tests
" we want to assess the product quickly
" we want to explore areas of the system that are unclear
" creative skills in testing are being encouraged
" we don’t know all the detail of the requirements
ET is useful when…
28
© Lloyd Roden Testing Requirements 131014
testing requirements-15
Summary
" various forms of requirements exist
" there are many challenges that we are often faced with when testing requirements
" testing must rise to these challenges by using a variety of techniques, methods and tools in order to be as effective as we can in testing the requirements
29