ambiguity_reviews_improving_bus_reqs_schanta_stareast2014
TRANSCRIPT
| © 2014, Cognizant 0 © 2014, Cognizant | All rights reserved. The information contained herein is subject to change without notice.
Susan Schanta
Process, Quality & Consulting
April, 2014
Ambiguity Reviews
Building Quality Requirements
| © 2014, Cognizant 1
Agenda
The Cost of Quality 2
What is an Ambiguity Review? 3
Ambiguity Review Place in the SDLC 11
Ambiguity Review Classification 14
Ambiguity Review Types 20
Ambiguity Review Template 23
Ambiguity Review – Keyword Searches 27
Ambiguity Review – Content Review 30
Ambiguity Review Metrics 35
Ambiguity Review – Benefit to QA 40
Ambiguity Review – Be Nice About It! 42
Appendix 44
| © 2014, Cognizant 2
The Cost of Quality
2
• IRS Tax Systems modernization project spent $3.3b before
canceling (Federal Computer Week March, 2002)
• Time Warner Communications spent $1b on a failed
information system to break into the Residential Telephone
Business (Computerworld May, 1997)
• Sainsbury, a British food retailer wrote off $526m
invested in an Automated Supply-Chain Management
System.
• Ford Motor Company spent $400m on a Purchasing
System before abandoning it in 2004
50% of defects are due to requirements problems (Schwaber, 2006)
| © 2014, Cognizant 3
What is an Ambiguity Review?
| © 2014, Cognizant 4
What is an Ambiguity?
Ambiguity can be defined as…
1. Use of words that allow alternative interpretations
2. An unclear, indefinite, or equivocal word, expression,
meaning, etc.
3. The possibility of interpreting an expression in two or more
distinct ways
4. Doubtfulness or uncertainty of meaning or intention: to speak
with ambiguity; an ambiguity of manner. (dictionary.com)
| © 2014, Cognizant 5
What is an Ambiguity Review?
An Ambiguity Review is a formal review process that focuses
on the identification of ambiguities in the language, structure
and logic of a requirement.
Provides a measure of whether requirements are
quantitative, clear, correct and complete.
Eliminates requirements defects from being identified in
later phases by building quality into the product. (Bender)
| © 2014, Cognizant 6
What Are the Benefits of Ambiguity Reviews?
Ambiguity Benefits realized…
• Scope Creep is controlled
• Reduced cost of maintenance
• Reduced number of change requests
• Increased traceability from requirements to test cases
Defects per requirement
Tangible Results…
Healthcare Insurer
• Reduced Defect Leakage from 35% to 8%
• Introduced BA Style Guide to drive down ambiguities
Fortune 50 Insurance Company #1
• Reduced ambiguities from an avg. 7 per Req. to 2 per Req.
• Introduced BA Style Guide to drive down ambiguities
Fortune 50 Insurance Company #2
• 63% of all defects were traced to Requirement Defects
| © 2014, Cognizant 7
Business Users Focus on Happy Path
Impact to business users
• Who focus on Happy Path
Results identify…
• Open ended questions
What are your pain points?
• Requirements lack input/output, alternative flows, error
conditions and constraints
• Development makes assumptions to fills in gaps in the
requirements
• Potential Defect leakage to production
Why do an Ambiguity Review?
| © 2014, Cognizant 8
Development Drives Requirements
Impact to Development
• Who drive application behavior based on assumptions
where requirement gaps exist
Results identify…
• Higher maintenance is a higher percentage of IT budget
Limits dollars spent on revenue producing software
Increases redo work
• Change requests trend higher
• Quality Assurance is expected to test quality in
Why do an Ambiguity Review?
| © 2014, Cognizant 9
QA Builds In Quality, Right?
Impact to QA
• Who are expected to test quality into the product
Quality cannot be tested into the product…
Results identify…
• Test Cases may not address..
Gaps in user scenarios
Gaps in alternative flows (error conditions)
Gaps in business rules
• Confusion and delays with requirements often results in a
compressed schedule for testing
Increases redo work
• Defect leakage into production
Why do an Ambiguity Review?
| © 2014, Cognizant 10
Ambiguity Review Benefits
• Limit Scope Creep
• Increase Requirements Traceability to Test Cases
• Reduce Defect Leakage to Production
• Improves Estimation Accuracy
• Reduce Cost of Maintenance
− How much is your organization spending to maintain systems today?
• Reduces redo work
• Increases velocity of Test Case creation
• Increase BA productivity and work product
− Elicitation Checklists
− BA Style Guide
How do I measure success?
| © 2014, Cognizant 11
Ambiguity Review Place in the SDLC
| © 2014, Cognizant 12
Ambiguity Reviews in Requirements Phase
Requirements Phase
• Elicitation Preparation
• Elicitation Session
• Document Elicitation Results
• Confirm Elicitation Results
BA Peer Review
Ambiguity Review
Final Review & Signoff by Key Stakeholders
Note: Ambiguity Reviews are SDLC agnostic. The process
can be applied to any lifecycle and any document format.
Building Quality In…
| © 2014, Cognizant 13
Ambiguity Review in the Requirements Phase Where do Ambiguity Reviews fit into the Lifecycle?
| © 2014, Cognizant 14
Ambiguity Review Classification
| © 2014, Cognizant 15
Ambiguity of Reference & Ambiguous Statements
A condition when a requirement uses words such as pronouns,
adjectives, adverbs and verbs that can be interpreted differently
based on the reader’s view.
The report shall run frequently.
Ambiguity: What is the name of the report?
Ambiguity: What are the data elements in the report?
Ambiguity: Frequently is not measurable
Ambiguity: What is the report generation schedule?
Ambiguity: What happens if there is no data for the report?
Ambiguity: What happens if the report fails?
(Wiegers, Bender)
| © 2014, Cognizant 16
Boundary Ambiguity
A condition when the author uses terms - among or up to. The
scope of the requirement is ambiguous because the stated
requirement can be interpreted in multiple ways.
Example:
1. If an employee makes less than $20,000 per year, the
employer pays 100% of the healthcare premium.
2. If an employee makes more than $20,000 per year, the
employer pays 50% of the healthcare premium for the
employee.
Ambiguity: What if an employee makes exactly $20,000?
(Wiegers, Bender)
| © 2014, Cognizant 17
Built-In Assumptions
A condition when the author assumes that all consumers of the
document will have the same level of domain knowledge
• Industry domain knowledge
• Subject domain knowledge
• Functional knowledge
• Environmental knowledge
Example: The system must apply the same limitations to
searches for existing groups as currently exists in Google Search.
Ambiguity: The requirement assumes the reader knows how the
functionality exists today.
(Wiegers, Bender)
| © 2014, Cognizant 18
Dangling Else
A condition when a requirement states expected results (what
normally happens) but does not state exceptions and error conditions.
Dangling Else Can Shall
Could Should
Is one of Will
Must Would
Example: The employee address type shall be either house,
apartment or condominium.
Ambiguity: The requirement does not consider exception conditions
such as PO Box.
(Wiegers, Bender)
| © 2014, Cognizant 19
Etc.
Etcetera is not a quantifiable measurement that can be confirmed so
it is considered totally ambiguous. (Phrases or sentences ending with
etc…)
Example: Subscribers shall identify themselves with unique
information (policy number, social security, etc.) when they call
Customer Care for information about their policy.
(Wiegers, Bender)
| © 2014, Cognizant 20
Ambiguity Review Types
| © 2014, Cognizant 21
Categorizing Ambiguities to Support Metrics
Ambiguity Categorizations will help with…
• Tracking patterns of ambiguities
− For a Business Analyst
− For the Program
• Building Elicitation Checklists
− Lessons learned turned into questionnaires
− Mentoring sessions for the BA Team
• Team Performance Measurements
− Scope Management
− Defect Leakage to Production
− Requirement Defects
Purpose of Ambiguity Types
| © 2014, Cognizant 22
Ambiguity Type Description
Ambiguous Term Terms (Phrase or Word) used in requirements which can be interpreted
by the reader in multiple ways e.g. frequently, occasionally, efficiently
Conflicting Requirement
Requirements which contradict each other – either in the same
document or across multiple documents.
e.g. The field name is Effective Date but the data type is defined as an
integer
Glossary Word or acronym used in requirements that is new or not commonly
used but has not been defined in the Glossary/Definitions section.
Grammar, Spelling
& Wording
Grammar, spelling corrections and proposed wording improvements to
increase clarity of the requirement
Incomplete Requirement
Incomplete requirement or statement describing conditions when
information is not fully detailed preventing design or test validation
e.g. The system shall handle 15-25% increase in the second year
Missing Requirement
Missing requirements that were not documented or may not have been
elicited from the business user.
e.g. Missing requirements – alternative flows, business rules,
exceptions and error conditions (Questions – What, When, Where)
Unclear Requirement
Requirements or statements requiring further clarification to allow the
reader user to fully understand the requirement (Questions – How,
Why, What do you mean by…)
(Wiegers, Bender)
| © 2014, Cognizant 23
Ambiguity Review Template
| © 2014, Cognizant 24
Ambiguity Review Template
• Transfer requirements document to Excel template
− Revision History
− Introductory information (free text)
− Business Requirements
− Data presented in tables such as Glossary, Field Elements
and Financial Information segregated to its own worksheet
• Ambiguity Columns to the Right
− Ambiguity Type
− Ambiguity Description
Transform the Requirements Document to Excel
| © 2014, Cognizant 25
Requirements Worksheet Example Not all columns need to be transferred to the Ambiguity Template…
| © 2014, Cognizant 26
Glossary Worksheet Example Text transferred to Excel template…
| © 2014, Cognizant 27
Ambiguity Review – Keyword Searches
| © 2014, Cognizant 28
Ambiguity Review Keywords
An initial review for keywords helps to identify Incomplete
Requirements
All files are transmitted daily.
All 820 Payments Files shall be transmitted to the Federal and
Maryland Jurisdiction after the nightly batch jobs complete
• The Federal 820 Payment File shall be sent daily during the
transmission window between 11 PM and 4 AM EST
• The Maryland 820 Payment File shall be sent daily during the
transmission window between Midnight and 4 AM EST
• If no payments are made for a jurisdiction, an empty file shall
be sent to the jurisdiction
Clues to Finding Ambiguities…
| © 2014, Cognizant 29
Ambiguity of
Reference Ambiguous
Adjectives Ambiguous Adverbs Ambiguous Variables
Ambiguous
Verbs
above ordinary infrequently the database derive
below rare intuitively the field determine
it same just about the file edit
such seamless more often than not the frame enable
the previous several more or less the information improve
them similar mostly the message indicate
these some nearly the module manipulate
they standard normally the page match
this the complete not quite the rule maximize
those the entire often the screen may Ambiguous Adjectives transparent on the odd occasion the status might
all typical ordinarily the system minimize
any usual rarely the table modify
appropriate valid roughly the value optimize
custom Ambiguous Adverbs seamlessly the window perform
efficient accordingly seldom Ambiguous Verbs process
every almost similarly adjust produce
few approximately sometime alter provide
frequent by and large somewhat amend support
improved commonly transparently calculate update
infrequent customarily typically change validate
intuitive efficiently usually compare verify
invalid frequently Ambiguous Variables compute
many generally the application convert
most hardly ever the component create
normal in general the data customize
(Bender/Wiegers)
| © 2014, Cognizant 30
Ambiguity Review – Content Review
| © 2014, Cognizant 31
Ambiguity Review for Content
• Ambiguities in content focus on the following
− Conflicting Requirements
− Incomplete Requirements
− Missing Requirements
− Unclear Requirements
• Questions of Who, What, When, Where & Why
• Questions of How tend toward design questions
− Only use to clarify expected behavior or outcome
− Do not use to ask how the system will process the function
behind the scenes
Clues to Finding Ambiguities…
| © 2014, Cognizant 32
Sample Ambiguity Questions
• What is the name of the file?
• Who is the user?
• What permissions does the user need to review the file?
• When is the file sent?
• What are the contents of the file?
• Where is the file sent?
• What are the business rules to validate the file?
• What are the error messages displayed when a business rule
validation fails?
• What happens if the file is corrupted and can’t be read?
• What happens if the file fails to be generated?
• Is an alert sent if a file is late, corrupt or fails?
• Who is the alert sent to?
Who, What, When, Where, Why…
| © 2014, Cognizant 33
Ambiguity Review Example
If the policy is a subscriber + spouse/domestic partner +
dependents policy, subscriber wants to cancel from the policy,
Exchange will send us 834 file to enroll the spouse/ domestic
partner as the subscriber under a new SID and the dependents
will remain as dependents under the new subscriber.
Note: Subscriber + Family
Content Questions Raised for this Requirement
| © 2014, Cognizant 34
Ambiguity Questions Posed…
| © 2014, Cognizant 35
Ambiguity Review Metrics
| © 2014, Cognizant 36
Ambiguity Review Metrics Patterns within the Requirements Document
3
6
Top 10 Ambiguities by Type Total
Ambiguous Term 55
Grammar Spelling & Wording 79
Incomplete Requirements 59
Unclear Requirements 57
Total Ambiguities 250
| © 2014, Cognizant 37
Program Level Ambiguity Review Metrics Patterns across the Requirements Document
| © 2014, Cognizant 38
Program Level Ambiguity Review Metrics
Patterns across the Requirements Document
| © 2014, Cognizant 39
Program Level Ambiguity Review Metrics
Patterns across the Requirements Document
Ambiguity Review Summary Totals
Total Requirement Considered* 3,737
Total Ambiguous Observations 26,478
Top 10 Ambiguities 7,463
Percent Ambiguities in Top Ten 28%
Average Ambiguities Per Requirement 7*Requirements include Non-Requirement Sections such as Introduction, Purpose, etc.
| © 2014, Cognizant 40
Ambiguity Review – Benefit to QA
| © 2014, Cognizant 41
QA Benefits from Ambiguity Review
• Geographically dispersed teams benefit from the additional
level of detail documented in the requirements
• Test Case Creation is simplified by clear requirements
− Normal Flow & Alternative Flows detailed in Use Cases
− Error conditions with error messages
− Business rules and constraints
− Data requirements
− Expected Outcome
• Traceability to requirements provides QA Engineer with path
to use cases and functional/non functional requirements
− Diagnosis of defect types clear
• Requirements vs. Code vs. Test Case
Quantifiable Impact
| © 2014, Cognizant 42
Ambiguity Review – Be Nice About It!
| © 2014, Cognizant 43
Business Analysts Have Feelings
Ambiguity Reviews are meant to be objective and focus on the
requirements content. However… You are critiquing a
Business Analyst’s work.
• Recognize the Business Analyst’s work effort
• Recognize the Business Analyst’s challenges with eliciting
requirements from business users.
• Be courteous when providing feedback.
Write Ambiguity Questions that are quantitative, clear, correct
and complete!
Be respectful of your feedback
| © 2014, Cognizant 44
Appendix
| © 2014, Cognizant 45
Requirements Review Phase
| © 2014, Cognizant 46
References
Writing High Quality Requirements
By Karl E. Wiegers, 2006
The Ambiguity Review Process
By Richard Bender
Assessing the Impact of Poor Requirements on Companies
By Keith Ellis
Executive Guide to Evaluating Business Requirements Quality
By Keith Ellis
Getting Consensus on Business Requirements – Tips and Traps
By Keith Ellis
The Quest for Good Requirements
By Dr. Martin Schedlbauer, 2011
How to Prevent the Negative Impacts of Poor Requirements
By Sergey Korban, April 30, 2013
The Business Value of Better Requirements
By Karl E. Wiegers, 2006