dan galorath
TRANSCRIPT
Estimation & Planning Processes Decide Project SuccessNASA Project Management Challenge 2012
Dan Galorath, Founder & CEO
Copyright Galorath Incorporated 2011
Introduction
• Estimating is critical for all kinds of systems
• Yet many treat is as a second rate process
• Having a repeatable estimation process is critical to both estimating AND to successful projects
• Estimation and measurement go hand in hand
© 2009 Copyright Galorath Incorporated 2
Delusions of Success: How Optimism Undermines Executives' Decisions (Source: Richard Hartley, HBR)
• Problem:
Humans seem hardwired to be optimists
• Optimism from cognitive biases & organizational pressures• Exaggerate talents & degree of control
• Attribute negative consequences to external factors
• Anchoring (relying too heavily on one piece of information) magnifies optimism• Most pronounced for new initiativesSolution: Temper with “outside view”
Supplement traditional forecasting w/ statistical Parametrics
Don’t remove optimism, but balance optimism & realism
Example of Tempering: (Source How To Measure Anything)
• German Mark V Tanks
• Allies estimated production by analyzing serial numbers
• Used the captured tanks as a random sample and predicted probability of various production levels
An Estimate Defined
• An estimate is the most knowledgeable statement you can make at a particular point in time regarding:• Effort / Cost
• Schedule
• Staffing
• Risk
• Reliability
• Estimates more precise with progress
• A WELL FORMED ESTIMATE IS A DISTRIBUTION
5
Estimation Methods 1 of 2
Model Category
Description Advantages Limitations
Guessing (Common)
Off the cuff estimatesQuick Can obtain any answer desired
No Basis or substantiationNo ProcessUsually WrongGenerally optimistic
Analogy (Common)
Compare project with past similar projects.
Estimates are based on actual experience.
Truly similar projects must exist.Less optimistic
Expert Judgment (Common)
Consult with one or more experts.
Little or no historical data is needed; good for new or unique projects.
Experts tend to be biased; knowledge level is sometimes questionable; may not be consistent.Generally optimistic
Top Down Estimation (Common)
A hierarchical decomposition of the system into progressively smaller components is used to estimate the size of a software component.
Provides an estimate linked to requirements and allows common libraries to size lower level components.
Need valid requirements. Difficult to track architecture; engineering bias may lead to underestimation.Generally optimistic (miss non-WBS items)
6
Estimation Methods 2 of 2
Model Category Description Advantages Limitations
Design To Cost (sometimes)
Uses expert judgment to determine how much functionality can be provided for given budget.
Easy to get under stakeholder number
Little or no engineering basis.Optimistic
Simple CER’s (common)
Equation with one or more unknowns that provides cost / schedule estimate
Some basis in data
Simple relationships may not tell the whole storyHistorical data may not tell the whole storyLess optimistic
Comprehensive Parametric Models (common)
Perform overall estimate using design parameters and mathematical algorithms.
Models are usually fast and easy to use, and useful early in a program; they are also objective and repeatable.Easy tradeoffs can provide better plans
Models can be inaccurate if not properly calibrated and validated; historical data may not be relevant to new programs; optimism in parameters may lead to underestimation. More or less optimistic depending on parameters
7
Why should we care: Each method has challenges and we should be familiar with each
Common Challenges In Estimating
• High effort and expenses in producing estimates Proposals - Acquisition Strategy inputs
• Informal request - POM budgets
• Significant investment in time to generate numbers
• Weak accuracy in numbers leading to cost overruns
• Customer frequent request for revisions cause even most expenses
• Numbers susceptible to many forces without robust cost generation systems
• Lack of understanding of cost risk
• Difficulty in calibrating “Engineering Judgment
© 2009 Copyright Galorath Incorporated 8
Poor Estimates Effect on Projects• Inaccurate estimates can reduce project success:
• Poor implementations
• Critical processes don’t scale
• Emergency staffing
• Cost overruns caused by underestimating project needs
• Scope creep• Forever changing project goals
• Frustration
• Customer dissatisfaction
• Cost overruns and missed schedules
• Project Failures
• Important project business decisions made early with minimum knowledge & maximum uncertainty
9
Why should we care: Poor estimates and plans are a root cause of program failure
Initial Cost Estimation Problems (Software Program Managers Network)
• Many programs that have been evaluated tend to initially estimate using a very optimistic method.
• Low bid to win contract
• Naïve level of effort estimation
• Human optimism (HBR & Rich Hartley USAF Undersec Acquisition)
• Often wrong since they are not based on a thorough analysis of requirements. The formula
Cost = Size x Complexity/Productivity • Has three unknowns upfront: size, complexity, and productivity.
• Methods & estimating tools determine size given system requirements
• Organizational databases of productivity on comparable size and complexity projects can be used
• Or other bottom-up estimation techniques
“In any event, all initial cost estimates should be considered as potential high risk and should be reviewed at each program review” SPMN
Software Is A Critical Component of Military & Aerospace Systems Failures Ariane 5
• Ariane 5 video
• Note: $500 Million payload
• Failure due to reused software from (Ariane 4)
• with incompatible assumptions for exception condition that was not required
ariane_5_explosion_video.mp4
“the culture within the Ariane program…” only addressed “random hardware failures… which can quite rationally be handled by a backup system”“the view had been taken that software should be considered correct until it is shown to be at fault”!
Why should we care: Software cost & schedule should be sufficient for successful missions
Software Is A Key Risk Item In Weapons Systems
• Navy Mobile User Objective Satellite Communication System delays to the Joint Tactical Radio System, a set of software-defined radios causes advanced MUOS capabilities to drastically underused… GAO
• GAO identified 42 programs at risk for cost & schedule 1. military requirements changes
2. software development challenges
3. workforce issues
• National Institute of Standards and Technology (NIST)
• Software defects cost nearly $60 Billion Annually
• 80% of development costs involve identifying and correcting defects
Software, not Hardware or technology readiness levels were called out
Death March Projects Are Likely To Fail
Why should we care: If you have a project on a death march failure is highly probably
Balancing Resources & Schedule Is A Science
For a given Size, Complexity and Technology
Impo
ssib
le
Minimum TimeTo Complete
(Effort Increases
to Reduce Schedule)
Work ExpandsTo Fill Time
(Effort Increases
due to lack of pressure)
Inefficient
Minimum Time
Optimal Effort(Lower Effort for Longer Schedule)
Calendar Time
Effo
rt M
onth
s
Effort Increasedue to Longer Schedule
Why Total Cost Grew for Two Space ProgramsDavid Graham, NASA
5 “The Success Triangle of Cost, Schedule, and Performance: A Blueprint for Development of Large-Scale Systems in an Increasingly Complex Environment” - (Booz|Allen|Hamilton, 2003)
Development Growth Causes
25%
11%
25%
9%
11%
11%
8%
RequirementsGeneration & Translation
Budget/Funding
Cost Estimation
Underestimation of Risk
Schedule Slips (Govt &Contractor)
Price Increases
Other
Quantitative Framework
Use Earned Value To Quantify Progress Versus Effort (Oversimplified)
• Main EVM concern is what has been accomplished in a given time and budget, versus what was planned for the same time and budget• Project generally deemed healthy if what has been
accomplished is what was planned, or more
• Project deemed unhealthy if accomplishment lags expectations
• Definition: Earned value = budgeted value for the work accomplished (what you got for what it cost you)
16
Time = Now
Budget$
EV
HealthyBudget$
Time = Now
EV
Unhealthy
Deploying Before Complete Leads To Program Disasters: You Better Understand Schedule
17
0 4 8 12 16 20
Schedule ProbabilityExample Application 1
Probability
Time (calendar months)
1%10%20%30%40%50%60%70%80%90%99%
Understand Project Risks Include Them In Planning Decisions (Example SEER-SEM Outputs)
0 1800 3600 5400 7200 9000
Effort (person-hours)
1%10%20%30%40%50%60%70%80%90%99%
Effort ProbabilityExample Application 1
Probability
0 12 24 36 48 60
Defects ProbabilityExample Application 1Probability
Defects (count)
1%10%20%30%40%50%60%70%80%90%99%
18
Considering Maintenance During Planning Can Yield More Successful Long Term Results
Staff Vs Maintenance Rigor
0500
100015002000250030003500
1 7 13 19 25 31 37 43 49 55 61 67 73 79 85
Time
staf
f hou
rs p
er m
onth
develop
Rigor vhi+
Rigor nom
Rigor vlo
19
Data Doesn’t Have To Be Perfect To Be Useful: But Is Has To Be Viable
• 80 Calories per serving
• 2.5 Servings per can
• 4 Ounces, Condensed, 8 Ounces With Water
You have an estimate …Now what?
21
• Correlation does not imply causation• Just because two data points may sit side by side
doesn’t mean they are the same or will have the same outcome
• Casual analysis is a recognized error in medicine
Perhaps ???
The Error of Causal AnalysisCreating a False Association
Tumor Can Cause Headache
Headache doesn’t mean a tumor
22
Use Historical Measurement to evaluate your estimate!
It’s easy to dig deeper and deeper to justify an estimate!23
24
Data Beware Apples and Oranges:
Phase : All activities may not be included.
System Concept missing
Integration missing
Phases
System Concepts System Req & Design
System Req Analysis Preliminary Design
Detailed Design Code / Unit Testing
Software Test System Integration / OT&E
Estimation Process
25
26
Basic Cost Estimating Process (Source CEBOK)
• Work Breakdown Structure (WBS) Development
• Program/System Baseline Development
WBS
Baseline
Data Collectio
nData
Analysis
Methodology
Validation
Reports
27
US GAO process for Credible Estimates
28
10 Step System Estimation Process 2011
1. Establish Estimate Scope
2. Establish Technical Baseline, Ground
Rules, Assumptions
4. Refine Technical Baseline Into
Estimable Components
4. Collect data / estimation inputs
5. Estimate Baseline Cost, Schedule, Affordability Value
6. Validate Business Case Costs &
Benefits (go / no go)
6. Quantify Risks and Risk Analysis
8. Generate a Project Plan
9. Document Estimates and Lessons
Learned
10. Track Project Throughout
Development
Estimation Organizational Maturity V1.7Level 0 Informal or no estimating Manual effort estimating without a process
Level 1 Direct Task Estimation Spreadsheets Ad Hoc Process
Level 2Formal Sizing (e.g. function
points)
Direct Task Estimation
Simple model (Size * Productivity) or
informal SEER Use
Some measurement &
analysisInformal Process
Level 3 Formal Sizing
Robust Parametric estimation
(SEER)
Estimate vs. actual capture
Formalized Multiple Estimate Process
Rigorous measurement & analysis
Parametric planning & Control
Risk Managem
ent
Repeatable
process
Level 4 Formal sizing
Repeatable process
Robust parametric estimating
(SEER)
Rigorous measurement & analysis
Parametric estimation with
tracking & control
Risk Managemen
t
Process improvement via lessons
learned
Level 5 Formal sizing
Repeatable process
Robust parametric estimating
(SEER)
Rigorous measurement & analysis
Parametric estimation with
tracking & control
Risk Managemen
t
Continuous process
improvement
Why should we care? Maturity is related to estimate viability… With better estimation process, projects more likely to be successful in execution
Estimation Should Be Key Part of Process (Source Q Redman, APMP Just Say No)
30
Scop
e &
Accu
racy
Us e
1 2 3 4Phase 0-1
ROM
ROM
ROM
ROM
AcquisitionPlanning/ POMand Plus Ups
Market Assessment/
“What If’s”
OpportunityCreation/ Customer
Decision PlansProcurement
Initiation Draft RFP RFP
DDE
Modified Budgetary Estimate
Draft RFP/Gate 36-8 people, 3
weeks(Bid Stds + History
)
Formal BidGate 4
15-20 people4 weeks
(Bid Stds+ History)
EARLY ESTIMATING3-5 people, 3 - 5 days
Top Down, parametric model based price estimating
Vs.Current state: 90 people, 6wks
31
• This chapter discusses a 1972 GAO report on cost estimating• We reported that cost estimates were understated and causing unexpected cost growth
• Many of the factors causing this problem are still relevant today
• We also discuss a 12 step process for producing high quality cost estimates
GAO Publication: Characteristics of credible cost estimates and a reliable process for creating them
32
GAO Publication: Why cost estimates are required for government programs and challenges developing credible results
• Introduces why cost estimates are required for government programs
• Developing annual budgets, supporting management decisions about which program to fund, and evaluating resource requirements at key decision points
• Discusses various challenges associated with developing credible results
Ask These Questions When Identifying Estimate Scope• Challenged projects
• Would you still go forward if you knew• Schedule would be significantly longer?
• Cost would be dramatically higher?
• Probably: but perhaps more insight could identify mitigation
• Plan functionality differently
• Certainly you could notify stakeholders of real costs
• Ensure staffing is appropriate for the constraints
• Failed Projects
• Would you start a project you knew was unaffordable? Or if schedule was completely unrealistic?
• If knowing up-front could you do something about it?
• Often better to kill project before it begins than waste resources & let the organization down
33
Lesson Learned: Estimate Must Quantify Risk & Uncertainty
3434
What is likely to happen
Feel lucky?
Firm Fixed Price?
Understand the risk before you commit!
Estimation Best Practices
© 2011 Copyright Galorath Incorporated
35
36
Cost Estimate Qualities (Adapted from CEBOK)
• The characteristics of high quality cost estimates are:• Accurate (Viable Within Range)
• Comprehensive
• Replicable and Auditable
• Traceable
• Credible
• Timely
37
System Description (Parametrics Can Estimate More, Earlier) Adapted from CEBOK
“If you can’t tell me what it is, I can’t tell you what it costs.”
-Mike Jeffers
“If you can tell me the range of what it might be I can tell you the
range of cost, schedule & probability”
-Dan Galorath
38
Types of Cost Estimates (Adapted From CEBOK)
• Life Cycle Cost Estimate (LCCE)
• Independent Cost Estimate (ICE)
• Budget Estimate
• Rough Order of Magnitude (ROM)
• Estimate At Completion (EAC)
• Independent Cost Assessment (ICA)
• Analysis of Alternatives (AoA)
• Economic Analysis (EA)
1
39
Aircraft Example: Estimating Techniques Adapted from CEBOK)
• Where applicable, use primary methods
• Analogy• Model X100 series jet engines have only been used on one other
plane, but weight is 100% higher on this model; estimate 2x other model
• Simple Parametric• As shown, need to estimate 2 lb brake rotors, should be roughly $4M from regression curve
• Commercial Parametric• Key characteristics range are
• 30-50k lines of code and 600-800
• kilos engine & 7 PC boards
• Engineering Build-Up• Know Air Conditioning (AC) system costs on plane because received
quotes from HVAC vendor for all duct work and AC blower off the shelf
Parametric Estimate
0
2
4
6
8
10
12
0 2 4 6 8 10
Parameter (ie. w eight)
Co
st
Manual Estimates: Human Reasons For Error (Metrics Can Help)
• Manual Task estimates yield SIGNIFICANT error
• Desire for “credibility” motivates overestimate behavior (80% probability?)• So must spend all the time to be “reliable”
• Better approach: force 50% probability & have “buffer” for overruns
• Technical pride sometimes causes underestimates
40
41
Lesson Learned: The Risk In Risk Analysis
"It's tough to make predictions, especially about the future." -- Yogi Berra.
42
Cost Readiness Levels at Low TRLs and/or Less Than Firm Requirements
• If the project has critical items at less than TRL 4…• Like asking Edison in 1876 “How
much longer for the light bulb”
• “Hard to say”, he would have said as he had you thrown out
• Note that this is not the same as asking, in 1879, once he had found a workable carbon filament, “How much will a production version of the light bulb cost to develop and produce Tom?”
• This would have been a TRL 4 question
• Tom’s cost estimators could have begun to model this
• So if 1<TRL<3 CRL < 4
• Likewise, if requirements are not firm, CRL < 4
TRL 9
TRL 8
TRL 7
TTRRLL 66
TTRRLL 55
TRL 4
TRL 3
TRL 2
TRL 1
TRL9: Actual system “flight proven” thorough successful mission operations TRL8: Actual system completed and “flight qualified” through test and demonstration TRL7: System prototype demonstration in a space environment TRL6: System/subsystem model or prototype demonstration in a relevant environment TRL5: Component and/or breadboard validation in relevant environment TRL4: Component and/or breadboard validation in laboratory environment TRL3: Analytical and experimental critical function and/or characteristic proof-of-concept TRL2: Technology concept and/or application formulated TRL1: Basic principles observed and reported
43
Table 1: CRL Rating Prior to Availability of Probabilistic Risk Analysis
CRL9 5 5.5 6 6.5 7 7.5 8 8.5 9
CRL8 4.5 5 5.5 6 6.5 7 7.5 8 8.5
CRL7 4 4.5 5 5.5 6 6.5 7 7.5 8
CRL6 3.5 4 4.5 5 5.5 6 6.5 7 7.5
CRL5 3 3.5 4 4.5 5 5.5 6 6.5 7
CRL4 2.5 3 3.5 4 4.5 5 5.5 6 6.5
CRL3 2 2.5 3 3.5 4 4.5 5 5.5 6
CRL2 1.5 2 2.5 3 3.5 4 4.5 5 5.5
CRL1 1 1.5 2 2.5 3 3.5 4 4.5 5
Extremel Minimal
Very Very Very Very Very Very Very Very
Basic “Complexity*”
of the to go work at the time
of the estimate
Adequacy of “Estimating Methods”, experience of the estimators, quality of CARD, availability of analogous data and
cost tools, time allowed for estimate, independence of estimate, number of
cross checks performed
*Complexity considerations include human rating, launch system requirements, planetary destination, operational vs experimental requirements, materials complexity, use of deployables, parts count, challenging thermal requirements, high data rates, electronic parts class, stabilization requirements, power generation type, propellant choice, propulsion requirements and many other factors. Programmatic complexity includes team size, team experience, schedule and many other factors.
CRL Description
9End of project actual cost
8Cost fit for very firm
engineering decisions and very firm budget
commitments (+/-5%)
7Cost fit for firm
engineering decisions and firm budget commitments
(+/-15%)
6Cost fit for PDR
engineering decisions and PDR budget use
(+/-25%)
5Cost fit for preliminary
engineering decisions and preliminary budget use (+/-
35%)
4Cost fit for very
preliminary engineering decisions and very
preliminary budget use (+/-45%)
44
Table 2: CRL Rating After Availability of Probabilistic Risk Analysis
25th Percentile Cost
Median Cost 75th Percentile CostLookup
Ratio of 75th Percentile Cost to 25th
Percentile Cost
ReadCRL
Description
100 100 100 1.00 9End of project actual cost
95 100 105 1.11 8Cost fit for very firm
engineering decisions and very firm budget commitments
(+/-5%)
85 100 115 1.35 7Cost fit for firm engineering decisions and firm budget
commitments (+/-15%)
75 100 125 1.67 6Cost fit for PDR engineering
decisions and PDR budget use (+/-25%)
65 100 135 2.08 5Cost fit for preliminary
engineering decisions and preliminary budget use (+/-35%)
55 100 145 2.64 4Cost fit for very preliminary
engineering decisions and very preliminary budget use (+/-45%)
Use S Curve inter-quartile cost range to translate to a CRL rating– Calculate ratio of 75th percentile cost to 25th percentile cost; then lookup ratio on chart to read CRL
Estimation Best Practices
• Decide Why You Want An Estimate
• Map Estimation Goals To Estimate Process Maturity & Develop Plan To Achieve The Maturity
• Have A Documented, Repeatable Estimation Process
• Make The Estimating Process As Simple As Possible; But No Simpler
• Be Proactive: The Process Is Important, The Tools Go Along With The Process
• Get Buy-in From Program Managers
• Hold People Accountable: Center Of Excellence Can Prepare Estimate But Program Managers Must Own Them
• Tie The Estimate To The Plan
45
Estimation Best Practices 2
• Evaluate Total Ownership Cost; Not Just Development
• Estimate A Range And Pick A Point For The Plan
• Re-estimate The Program When It Changes
• Avoid Death Marches: Programs With Unachievable Schedules Are Likely To Fail And Drain Morale
• Keep A History: Start An Enterprise Database NOW…
• Business Case: Evaluate ROI In Addition To Costs
• Convert Expert Spreadsheets Into A Common Language
46
Estimation Best Practices 3
• Track Progress Vs. Estimate Throughout The Life Cycle
• Estimate Schedule As Well As Effort (Cost) For Complete Picture
• Tie The Business Case Into The Estimating Process
• Attack Non-productive Rework As Part Of The Process
47
Estimation Best Practices 4
• Have clear definitions:
• What does “complete” mean
• What activities are included and excluded (E.g. development only or total ownership; help desk included or excluded, etc.)
• Which labor categories are included and excluded in the estimate (e.g. are managers included? Help desk? Etc.)
• Don’t ignore IT infrastructure and IT services costs
• Tracking defect sources can go along with the process
48
A1-49
Project Management Challenges Relate To Estimation Planning• “No good deed will go unpunished.” Unreasonable
expectations on the next project are supported by heroic behavior on the previous project
• The most important business decisions about a software project are made at the time of minimum knowledge and maximum uncertainty.
• Adding and/or changing means more time and/or more effort
• When a project is in trouble ask for more time rather than more people. Deferring functionality common approach to asking for more time
• Increasing concurrency is usually not a solution (e.g. designing several releases concurrently)
50
7 Characteristics of Dysfunctional Software Projects (Source: Mike Evans, et al.)
• Based on 350 projects:
• Failure to Apply Essential Project Management Practices
• Unwarranted Optimism and Unrealistic Management Expectations
• Failure to Implement Effective Software Processes
• Premature Victory Declarations
• Lack of Program Management Leadership
• Untimely Decision-Making
• Lack of Proactive Risk Management
Additional Information
• www.galorath.com
• Dan on estimating BLOG: www.galorath.com/wp
• Email: [email protected]
© 2009 Copyright Galorath Incorporated 51