1
E S T I M A T E
Maximising Capital Investments:IS GUESSWORK ERODING YOUR BOTTOM LINE? E S T I M AT I O N B I A S A N D M I T I G AT I O N 2 0 1 7 E D I T I O N
Dan Galorath for Unico Enterprise Services
© 2016 Copyright Galorath Incorporated
Contact Details: Michael McKeon Unico Enterprise Services +61429054486 [email protected]
E S T I M AT E • A N A LY Z E • P L A N • C O N T R O L
2
Key Points
Every forecast is subject to estimation bias… a major cause
of program failure and corporate misspending.
Experts Bias is probably costing - in cost,
schedule, and less than hoped for benefits.
When bias is managed better decisions
are made and more successes follow.
3
Your People Are Optimism Biased
HARVARD BUSINESS REVIEW EXPLAINS THIS NOBEL PRIZE WINNING PHENOMENON:
- Humans seem hardwired to be optimists - Routinely exaggerate benefits and discount costs
- Bias permeates opinions & decisions & causes waste & failure Delusions of Success: How Optimism Undermines Executives’ Decisions (Source: HBR Articles | Dan Lovallo, Daniel Kahneman | Jul 01, 2003)
Solution: Temper with “outside view”: past measurement Results, traditional forecasting, risk analysis and statistical parametrics can help. Don’t remove optimism, but balance optimism and realism.
4
IT Failure Can Impact Business Dramatically
CASE STUDY: LEVI STRAUSS
• $5M ERP deployment contracted• Risks seemed small• Difficulty interfacing with customer’s systems• Had to shut down production• Unable to fill orders for 3 weeks• $192.5M charge against earnings on a $5M IT project failure
“IT projects touch so many aspects of organization they pose a new singular risk”
5
Cognitive Bias: tendency to make systematic decisions based on PERCEPTIONS
rather than evidence.
“Perception has more to do with our desires—with how we want to view ourselves—
than with reality.” Behavioral Economist Dan Ariely
Researchers theorize in the past, biases helped survival.
Our brains using shortcuts (heuristics) that sometimes
provide irrational conclusions.
Bias affects everything:
- From deciding how to handle our money
- To relating to other people - How we form memories
Bias Guesses & Forecasts SOURCE: BEINGHUMAN.ORG
6
The Planning Fallacy (KAHNEMAN & TVERSKY, 1979)
• Manifesting bias rather than confusion
• Judgement errors made by experts and laypeople alike
• Errors continue when estimators are aware of their nature
JUDGEMENT ERRORS ARE SYSTEMATIC AND PREDICTABLE, NOT RANDOM
OPTIMISTIC DUE TO OVERCONFIDENCE IGNORING UNCERTAINTY
ROOT CAUSE: EACH NEW VENTURE VIEWED AS UNIQUE
• Underestimate costs, schedule, risks
• Overestimate benefits of the same actions
• “inside view” focusing on components rather than outcomes of similar completed actions
• FACT: Typically, a past more similar assumed
• Even ventures may appear entirely different
7
Bias Mitigation Reference Class Forecasting PREDICTS OUTCOME OF PLANNED ACTION BASED ON ACTUAL OUTCOMES IN A REFERENCE CLASS: SIMILAR ACTIONS TO THAT BEING FORECAST.
Attempt to force the outside view and
eliminate optimism and misrepresentation
Choose relevant “reference class”
completed analogous projects
Compute probability distribution
Compare range of new projects to completed
projects
Best predictor of performance is actual performance of implemented comparable projects (Nobel Prize Economics 2002)
8
Bias Mitigation Reference Class Forecasting PREDICTS OUTCOME OF PLANNED ACTION BASED ON ACTUAL OUTCOMES IN A REFERENCE CLASS: SIMILAR ACTIONS TO THAT BEING FORECAST.
Attempt to force the outside view and
eliminate optimism and misrepresentation
Choose relevant “reference class”
completed analogous projects
Compute probability distribution
Compare range of new projects to completed
projects
Provide an “outside view” focus
on outcomes of analogous projects
Best predictor of performance is actual performance of implemented comparable projects (Nobel Prize Economics 2002)
10
E X P E R I M E N T :
Anchoring Biases EstimatesSOURCE: MYWEB.LIU.EDU/~UROY/ECO23PSY23/PPT/04-ANCHORING.PPTX
Result:Those who got higher numbers on the wheel of fortune guessed bigger numbers in Step 3
123
Subject witnesses the number that comes up when a wheel of fortune is spun
Is asked whether the number of African countries in the U.N. is greater than or less than the number on the wheel of fortune
Is asked to guess the number of African countries in the U.N.
If given a number, that biases estimates
11
Flaw of Averages Case Studies
(SOURCE: HBR)
• U.S. Weather Service forecast that North Dakota’s rising Red River would crest at 49 feet.
• Made flood management plans based on this average figure
• In fact, the river crested above 50 feet, breaching the dikes, and unleashing a flood that forced 50,000 people from their homes.
EXAMPLE: $2 BILLION PROPERTY DAMAGE IN NORTH DAKOTA
13
A G I L E :
Detailed Software Development Life Cycle Management
(SCRUM EXAMPLE)
- Focus is on what features can be delivered per iteration- Not fully defined what functionality will be delivered at the end?
- Iterations are often called a “Sprint”
14
Agile Is Not a Silver BulletSOURCE: HTTP://WWW.JAMASOFTWARE.COM/BLOG/RETHINK-AGILE-MANIFESTO-PROJECTS-STILL-FAIL/
- Projects still fail at roughly the same rate as 2001- Dr. Dobbs: Agile is not a productivity revolution
0%
20%
40%
60%
80%
Agile (72%) Traditional (64%)8% Improvementnot Revolution
Agile 8% More Success
15
Software Project Performance Still Disappointing
(McKinsey)
-80%
-60%
-40%
-20%
0%
20%
40%
60%
45% Over Budget 56% less Functionality thanplanned
McKinsey Study: Projects over $15m
So does skipping estimating solve this? Well: If commitments aren’t made there can be no disappointment
16
Estimation & Planning A COMPLETE ESTIMATE DEFINED
An estimate is the most knowledgeable statement you can make at a particular point in time regarding:• Effort / Cost• Schedule• Staffing• Risk• Reliability
• Estimates more precise with progress
• A WELL FORMED ESTIMATE IS A DISTRIBUTION
17
If A Promise or Hope Is Good Enough You Don’t Need Viable Estimates
NO NEED TO ESTIMATE IF:
• A promise of 5 sprints with 4 people is good enough• You are willing to spend whatever it costs in whatever time it takes• Estimates don’t impact planning or decision processes• You don’t need to know the probability that it will be complete or when • You aren’t concerned if it overruns substantially• You aren’t concerned with failure and contingency• You don’t need to consider total ownership costs• You are ok if this is an overestimate and resources are not optimized• There will be no system testing on top of development• If lack of documentation is ok for maintenance
19
#noestimates is Viable For Detailed DevelopmentBUT SHOULD NOT ABDICATE BUSINESS ISSUES FOR SUBSTANTIAL DEVELOPMENTS
Business Case Evaluation of alternatives
Agile or Hybrid Agile Software Development
System Test (when appropriate)
Maintenance and Support
FOR SUBSTANTIAL SYSTEMS
Hybrid Agile: Requirements & Design
How Much? How Long? Ownership Cost Go / No Go
• Agile development = root level software development management…• Story point estimating is short term productivity management• It is not a business decision making process
20
Mitigate Planning Fallacy & Bias
Additional Historical Data
Trend line
Trend line +1 sigma
Trend line -1 sigma
Matches Platform/Application
Recently added
Used For Calibration
Current Estimate
1
10
100
1,000
10,000
100,000
1,000,000
Dev
elop
men
t Effo
rt H
ours
(Log
)1 10 100 1,000 10,000100,0001,000,00010,000,000
Effective Size (Log)
Development Effort Hours vs Effective Size
Additional Historical Data
Trend line
Trend line +1 sigma
Trend line -1 sigma
Matches Platform/Application
Recently added
Used For Calibration
Current Estimate
1
10
100
1,000
10,000
100,000
1,000,000
Dev
elop
men
t Effo
rt H
ours
(Log
)1 10 100 1,000 10,000100,0001,000,00010,000,000
Effective Size (Log)
Development Effort Hours vs Effective SizeEstimate Agile Projects With SEERTO UNDERSTAND WHAT COST / SCHEDULE TO EXPECT
21
STEP
1
2
3
4
5
6
7
8
9
10
EXPECTED
30
50
80
50
90
25
35
45
70
25
500
Adding Reality to EstimatesEXAMPLE – 2 (SOURCE SEI)
22
STEP
1
2
3
4
5
6
7
8
9
10
BEST
27
45
72
45
81
23
32
41
63
23
EXPECTED
30
50
80
50
90
25
35
45
70
25
500
WORST
75
125
200
125
225
63
88
113
175
63
Adding Reality to EstimatesEXAMPLE – 2 (SOURCE SEI)
What would you forecast the schedule duration to be now?
23
Monte Carlo Analysis TO QUANTIFY AND MITIGATE BIAS
90% confidence, project under
817 days
50% confidence, project will be under 731 days
Almost guaranteed to miss the 500 days
24
SEER: Risk Driven Estimates THE ENGINE FOR PROJECT EVALUATION
• SEER predicts outcomes
• SEER uses inputs to develop probability distributions
• The result is a probabilistic estimate
• SEER will predict a likely range of outcomes
• Monte Carlo provides project-level assessments of risk
Least, likely, and most inputs provide a range of cost and schedule outcomes
Confidence (probability) can be
set and displayed for any estimated item
25
Key Points
Every forecast is subject to estimation bias…a
major cause of program failure and corporate
mis-spending
Experts Bias is probably costing in cost, schedule,
and less than hoped for benefits
When bias is managed better decisions
are made and more successes follow
26
• FLYVBJERG, BENT, Curbing Optimism Bias and Strategic Misrepresentation in Planning: Reference Class Forecasting in Practice, European Planning Studies Vol 16 No 1, Jan 2008,
• Kahneman, Daniel, and Amos Tversky. “Prospect Theory: An Analysis of Decision under Risk.” Econometrica 47, no. 2, March 1979.
• Johnson, H. Thomas. Relevance Regained. Free Press.
• Mislick, Gregory K.; Nussbaum, Daniel A.. Cost Estimation: Methods and Tools (Wiley Series in Operations Research and Management Science) (p. 143). Wiley.
• Rose, Todd. The End of Average: How We Succeed in a World That Values Sameness, HarperCollins.
• Taleb, Nassim Nicholas. Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets (Incerto). Random House Publishing Group.
Bibliography (Partial)• Glen, Paul; McManus, Maria. The Geek Leader’s Handbook:
Essential Leadership Insight for People with Technical Backgrounds. Leading Geeks Press.
• Kogon, Kory; Merrill, Adam; Rinne, Leena. The 5 Choices: The Path to Extraordinary Productivity. Simon & Schuster.
• Patterson. Crucial Conversations Tools for Talking When Stakes Are High, Second Edition. McGraw-Hill.
• Kahneman, Daniel. Thinking, Fast and Slow. Farrar, Straus and Giroux.
• Hubbard, Douglas W.. The Failure of Risk Management: Why It’s Broken and How to Fix It . Wiley.
• Hubbard, Douglas W.. How to Measure Anything: Finding the Value of Intangibles in Business .
• Galorath, Evans, Software Sizing, Estimation, and Risk Management: When Performance is Measured Performance Improves, Auerbach Publications, 2007