1© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Establishing Release Quality Levels and Release Acceptance Tests
Insight Forum
April 21, 2009
2© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
• Founder/CEO of Enthiosys– Agile Product Management consulting – Customer needs, roadmaps, business model– Product management mentoring and training
• Agile product guy– VP Bus Dev (Aladdin), VP Eng &Product Dev’t
(Aurigin), VP Systems Eng (EDS Fleet Services)– Board of Agile Alliance
• Author, speaker, blogger– “Innovation Games”– “Beyond Software Architecture”– “Journey of the Software Professional”– agile PM blog at www.Enthiosys.com
About Luke Hohmann
3© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 3
Goals
1. Provide a framework for thinking about quality
2. Create better results by building the right quality
3. Practical advice tohelp you succeed What
did I forget
?
4© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Agenda
• Some discussion – what’s a release? quality?
• The iron triangle or the quality box?
• Release quality levels
• Release acceptance tests
NOTA
TOOLSTALK
5© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Discussions (5 min)
• What’s a release?
• What’s quality?
6© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
What do we offer? And why?
• We offer things to our market ecosystem (customers, analysts, etc.) to achieve goals– A release is offered to generate revenue– Lo-fi prototypes are offered to improve designs– Betas are offered to gain pre-release feedback
(and maybe generate revenue…)
• To be offered a thing has to have a “quality” that is suitable to helping us realize the goal
7© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
What is Quality?
• “Quality is conformance to user requirements.” —Phillip Crosby, Quality is Free (1980s)
• “Quality is the absence of defects that would make software stop completely or produce unacceptable results.”
—Capers Jones, Applied Software Measurement (1991)
• “Quality is achieving excellent levels of fitness for use, conformance to requirements, reliability, and maintainability.”
—Watts Humphrey, Managing the Software Process (1980s)
• “Quality is value to some person.”—Jerry Weinberg, Quality Software Management (1992)
8© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Is that enough?
• What about the manner in which we create “the thing”? What about the code?
• What if the code is just plain ugly?
• And if you’re using an Agile method, what if your code doesn’t pass the green bar?
9© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Quality is a Relationship
• High Performance is High Quality– to users who notice low performance.
• Elegant Coding is High Quality– to developers who place high value on the opinions of
their peers.
• Zero Defects is High Quality– to users who would be disturbed by those defects.
• Lots of Features is High Quality– to marketers who believe that features sell products.
©2008 Information Architects, Inc.
10© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 ©2008 Information Architects, Inc.
The Dichotomy of Quality
• Intrinsic or Extrinsic?– Does quality exist in the things we
observe or is it subjective, existing only in the eye of the observer?
“You take your analytic knife, put the point directly on the term Quality and just tap, not hard, gently, and the whole world splits, cleaves, right in two—hip and square, classic and romantic, technological and humanistic—and the split is clean.”
—Robert PirsigZen and the Art of Motorcycle Maintenance
(1974)
11© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Total Quality is therefore…
Total Quality
Extrinsic Quality = Value
Intrinsic Quality = Craftsmanship
©2008 Information Architects, Inc., Adapted and Extended
12© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Agenda
• Some discussion – what’s a release? quality?
• The iron triangle or the quality box?
• Release quality levels
• Release acceptance test
13© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Should the triangle be a box?
Scope
Cost Schedule
The Traditional Iron Triangle
Value(Extrinsic quality)
Quality(Intrinsic quality)
The Quality Box
Cost Schedule
14© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
It is a system
Scope
Cost Schedule
The Traditional Iron Triangle
Value(Extrinsic quality)
Quality(Intrinsic quality)
The Quality Box
Cost Schedule
15© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
What can affect the system?
• Flip-chart exercise
16© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Choosing to affect the system
Value(Extrinsic quality)
Quality(Intrinsic quality)
The Quality Box
Cost Schedule
Let’s hold these as fixed…
Extrinsic Quality• Which features present?• How well do the work?• How well have they been tested?• Do you have enough?• Is this release ready when I need it?
Intrinsic Quality• Craftsmanship of the code• Maintainability• Level of defects (works as specified)
17© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Agenda
• Some discussion – what’s a release? quality?
• The iron triangle or the quality box?
• Release quality levels
• Release acceptance tests
18© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Release quality level
• A predetermined quality level chosen by product management that sets expectations regarding the intrinsic and extrinsic quality of a release.
• Enables organizations to make confident decisions about their context
• Negotiated with development
19© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000 19
Example from VeriSign Managed Security ServicesRelease Level Description
Release Level 5 All functionality fully certified. Near-zero probability of high severity errors and a low probability that medium severity errors.
Release Level 4 All new and modified functionality fully certified, except where risk is deemed minimal (e.g. internal facing reports, low probability use cases).
Features not fully certified documented in the test plan and/or project charter. Strategic regression testing of existing functionality performed.
Low probability that high severity errors will be identified in production. Project carries a moderate probability that medium severity errors.
Release Level 3 All new and modified functionality at least partially certified. Features not fully certified documented in the test plan and/or project charter. Partial regression testing performed. Moderate probability for high severity errors. Higher probability for medium severity errors.
Release Level 2 Most new and/or modified functionality partially certified. Features not fully certified communicated to stakeholders. Partial regression testing performed. Higher probability that high and medium severity errors will be identified.
Release Level 1 New and/or modified features not certified by QA. Regression testing may or may not occur. Level 1 releases may be available only for demo or controlled access purposes. High risk for high severity errors.
©2007 VeriSign, Inc. Presented at Agile 2007 conference with Enthiosys
20© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Set each scale separately
Intrinsic Quality Extrinsic Quality
5 • Everything certified!• Lots of automation!
5 • Customers will rave about this!
4 • All new stuff tested by QA• Full regression testing
4 • Consistency very high• Simple and sophisticated tasks
3 • All new stuff tested by QA• Almost full regression testing
3 • Documented• Complete for basic tasks
2 • Most new stuff tested by QA• Partial regression tested
2 • Not documented• Complete for basic tasks
1 • Not tested by QA• Not regression tested
1 • Not documented • Possibly incomplete
21© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Lower bound for intrinsic quality is very high
• Leading edge enterprises employ technologies that can approach 99% cumulative defect removal rates.
• The norm for US firms is a cumulative defect removal rate of 75%.
• A cumulative defect removal rate of 95% on a project appears to be a nodal point where several other benefits accrue. For projects of similar size and type, these projects:– have the shortest schedules.– have the lowest quantity of effort in terms of person-months– have the highest levels of user satisfaction after release
—Capers Jones, Applied Software Measurement (1991)
©2008 Information Architects, Inc.
22© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Why is Intrinsic Quality so Important?
• The Impact of code quality on testing
• Error Location Dynamics
• Error Feedback Ratio
• Technical Debt
©2008 Information Architects, Inc.
23© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Impact of Code Quality on Testing
Development: 10 days,
4 people, 4 KLOC, 1 d/KLOC
Development: 10 days,
4 people, 4 KLOC, 15 d/KLOC
How long to test? Assume ½ day to find & fix per defect.
Test time= 2 days
Test time= 30 days
Outcome: no time to finish testing, technical debt increases!
©2008 Information Architects, Inc.
24© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
12108 6 4 2 0
0
8
16
24
32
40
48
56
64
Time
Errors Located
Difficult errorstake longer to find:
1 hr/d to 50 hr/d
Error Location Dynamics
©2008 Information Architects, Inc.
25© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
The time to finish removing errorsis critically dependent on the error feedback ratio. The three simulations differ only in their feedback ratios.A 20% difference in feedback ratio leads to an 88% difference in completion time, but the next 10% increase leads to a 112% increase.
The time to finish removing errorsis critically dependent on the error feedback ratio. The three simulations differ only in their feedback ratios.A 20% difference in feedback ratio leads to an 88% difference in completion time, but the next 10% increase leads to a 112% increase.
ERROR FEEDBACK: Errors put into a system when attempting to correct other faults.
ERROR FEEDBACK RATIO: The number of problems created per fix.
EFR = ERRORS CREATED / ERRORS RESOLVED
Error Feedback Ratio
©2008 Information Architects, Inc.
26© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Technical Debt
• Once on far right of curve, all choices are hard
• If nothing is done, it just gets worse
• In applications with high technical debt, estimating is nearly impossible
• Only 3 strategies– Do nothing, it gets worse– Replace, high cost/risk– Incremental refactoring,
commitment to invest 1 2 3 4 5 6 7 8
Years
Technical Debt
Co
st
of
Ch
ang
e (
Co
C)
ProductRelease
ActualCoC
Optimal CoC
Customer Responsiveness
27© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Agenda
• Some discussion – what’s a release? quality?
• The iron triangle or the quality box?
• Release quality levels
• Release acceptance tests
28© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Lots of Testing Options!
Customer Customer TestsTests
Business Business Intent
(Fit, Fitnesse)(Fit, Fitnesse)
UsabilityUsabilityTestingTesting
Exploratory Testing
ProgrammerProgrammer TestsTests
Design IntentDesign Intent
(xUnit)(xUnit)
PropertyPropertyTestingTesting
Response,Response, Security SecurityScaling,…Scaling,…
From Brian Marick From Brian Marick http://www.testing.com/cgi-bin/blog/2003/08/21#agile-testing-project-1http://www.testing.com/cgi-bin/blog/2003/08/21#agile-testing-project-1
Run at least once each development episode
Run each time a story scenario is completed Assess each iteration
Assess each iteration
29© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Release Acceptance Tests
• Tests owned and written by the product manager / product owner / customer to verify that a story is complete and correct.
• Should be (mostly) automated (more later)
• Popularized by Agile methods, useful everywhere!
30© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Benefits of Acceptance Tests
• Increase team confidence that the system is correct
• Help PMs think through requirements
• Informs developer estimates
• Concrete definition of completion
• Early involvement of QA
• Automated regression testing for free
31© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
ATs test the users’ experience
• UTs should test application logic in detailgenerally in isolation
• ATs should test: but not...– Interaction & Flow Usability– Performance Look & Feel– Error Handling– Security
32© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
ATDD Cycle
33© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Acceptance Tests are “Quirky”
• Written by PMs/Customers... But executed in code?– ATs require a customer meta-language generic enough to
capture requirements, abstract enough to be maintainable– Simple enough to be easily understood– They are interpreted by various frameworks– Requires external systems and “spike” development
• Automation tools dictate form/format of tests– http://www.fitnesse.org/– http://seleniumhq.org/– http://robotframework.org
34© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Fitnesse Structure
35© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Fitnesse test example
Each row is one test
Fitnesse supports many different testing table styles that can be organized into test suites.
36© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Writing good acceptance tests
• Tell a story of accomplishing a goal– Goal test title– Activities rows in your test suite
• Expect them to be stable, even if the underlying system is going through a lot of change
• Don’t reference domain objects
37© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Consider “Given-When-Then”
• Given some contextWhen something happensThen some behavior
• Given an innovation games planner with a scheduled party
When they cancel the partyThen send a cancellation email to the players
and the facilitator, a cancellation confirmationto the planner, and ensure that the game
playURL points to the “cancelled party” page
38© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Overcoming common challenges
• You don’t have to automate everything
• Get started, but don’t expect quick turns
• Help your product managers by writing a few
• Make AT writing a natural part of the conversation
39© 2009, Enthiosys Inc. All rights reserved. www.enthiosys.com or 650.528.4000
Thank You !!!
Luke Hohmann
Founder & CEO
Enthiosys, Inc.
615 National Ave., Ste 220
Mountain View, CA 94043
cell: (408) 529-0319
www.enthiosys.com
Innovation Through Understanding®