cost risk analysis
DESCRIPTION
Mort Anvari, CRB Office. Cost Risk Analysis. [email protected]. Outline. Definition of Risk and Uncertainty Analysis Methods of Risk Analysis Risk Facets Schedule Risk. Risk Assessment Definition. Purpose - PowerPoint PPT PresentationTRANSCRIPT
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
1
Cost Risk AnalysisCost Risk AnalysisCost Risk AnalysisCost Risk Analysis
Mort Anvari, CRB Office
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E) • Definition of Risk and Uncertainty Analysis
• Methods of Risk Analysis
• Risk Facets
• Schedule Risk
OutlineOutline OutlineOutline
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
3
Process Quantify the impact of the different Risk factors on cost Identify inter-relationships between these Risk factors Combine individual Risk factors into an overall estimate range Interpret and report results
Purpose
The purpose of a Risk Assessment is to provide Management and decision makers with a realistic range of possible outcomes, given a fixed set of baseline assumptions!
Risk Assessment DefinitionRisk Assessment DefinitionRisk Assessment DefinitionRisk Assessment Definition
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Risk and Uncertainty DefinitionsRisk and Uncertainty Definitions
Risk is the Occurrence of an Outcome Subject to a Known Pattern of Random Variation, i.e. Known-Unknown. ( Changes in those Technical Parameters that are Captured in the Estimating Methodology )
Uncertainty is the Occurrence of an Outcome Subject to Unknown Random Fluctuations, i.e. Unknown-Unknown. ( Changes in those Parameters that are not Part of the Estimating Methodology )
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Cost Risk & Uncertainty DefinitionCost Risk & Uncertainty Definition
Cost Risk and Uncertainty Analysis is Placing Probability Distribution Around a Point Estimate
Cost Risk Analysis Assumes that the Analyst has Correctly Identified the Factors (Parameters) that Influence Cost and Taken them into Account when Producing the Point Estimate
If the Point Estimate is not a function of Influencing Factors, Performing a Risk Analysis will be of Little Value
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Risk and Uncertainty AnalysisRisk and Uncertainty Analysis
Risk Analysis is a Requirement
All Cost Estimates for Major Defense Acquisitions Must be
Accompanied by a Formal Risk Analysis
“Defense Acquisition Management Policies and Procedure”
DoD 5000.2
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Reasons for Doing Risk Analysis Reasons for Doing Risk Analysis
Point Estimates are Almost Always Wrong, because
Changes in the Specifications
Changes in the Requirements
Optimistic Estimates of Advances in Technology
Funding Instability
Schedule Slippage
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Technical Programmatic Cost Schedule
Physical Properties Material Availability
Sensitivity to Assumptions
Degree of Concurrency
Material properties Skill Requirement Sensitivity to Technical Risk
Sensitivity to Technical Risk
Software Complexity Environmental Impact
Sensitivity to Programmatic Risk
Sensitivity to Programmatic Risk
Integration Interface Contractors Stability
Sensitivity to Schedule Risk
Sensitivity to Schedule Risk
Requirement Changes
Funding Profile Estimating Errors Number of Critical Path
Operational Environment
Political Advocacy Estimating Errors
Reasons for RiskReasons for Risk
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Point EstimatePoint Estimate
Method for Producing the Point Estimate Must be the Same as that for Estimating the Risk
In fact, the Point Estimate for Total Cost Should be a By-Product of the Cost Risk Analysis
Analyst Should not Obtain a Point Estimate and then Later Apply a Totally Different Method to Assess the Risk
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Cost Risk Analysis will not Reduce the Risk Inherent in a Program
Risk Analysis Help Program Managers Understand the Nature of the Risk Involved, and the Uncertainty Associated with Cost Estimates
Risk Analysis result in a more Realistic Assessment of the Funding Required and Likelihood of Exceeding the Point Estimate
Risk ReductionRisk Reduction
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
11
Schedule
Cost Estimating
Technical
Programmatic
Cost Risk FacetsCost Risk Facets
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Risk ConceptsRisk Concepts
Severity of Consequence
Pro
babi
lity
of O
ccur
renc
e High RiskLow Risk
Low RiskIncreasing
Moderate Risk
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
When to do Risk AnalysisWhen to do Risk Analysis
Risk analysis reduces the uncertainty between requirements and funding
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Risk Analysis MethodsRisk Analysis Methods
Qualitative Methods (e.g., Subjective Assessments of Low, Medium, or High Risk) are of Most Use when There is Little or no Historical Data Available or Firm Requirements Have not Yet been Established
Quantitative Methods are Considered where Probability Distribution on Cost Elements or Drivers can be Estimated from Historical Data or Deduced from Expert Opinion
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Quantitative MethodsQuantitative Methods
Analytical Method involves the Mathematical Determination of a Total Cost Distribution from its Components Cost Distributions
Simulation Method Involves the Computer Generation of Random Costs from Component Distributions and Aggregation into a Total Cost Distribution
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Other MethodsOther Methods
Prediction Intervals, like stating “There is X% Chance that total cost will fall between Y and Z”
Predication Intervals are too gross a summary measure of the distribution of total cost
They are not needed if the entire distribution can be estimated
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Risk Analysis ApproachesRisk Analysis Approaches
Detail
Effo
rt
• Detailed Network & Stochastic
• Detailed Monte Carlo Simulation (each WBS)
• Bottom Line Monte Carlo Simulation
• Add a Risk Factor/Percentage
• Discrete Technical, Schedule, and Estimating Risks
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Ways to Perform Risk AnalysisWays to Perform Risk Analysis
CER:
Use Regression Analysis to Relate Cost Drivers to Historical Costs
WBS:
Apply Probability Distribution to Cost Elements and Aggregate them into a total cost
Stochastic:
Break Down the Acquisition and Management Processes into their Components Activities and Events in Proper Time Sequence Networks
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Probability DistributionsProbability Distributions
WBS Cost Distribution Total Cost
P.E.
P.E.
P.E.
+
+
+ Many More
=
Sum PE MLC (Mode)
CDF
Con
fide
nce
50%
20%
Fre
quen
cy
Sum PE MLC
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Combining Risk DistributionsCombining Risk Distributions
P.E.
P.E.
P.E.
+
+
+ Many More
CDF
PDFP.E.
P.E.
P.E.
+
+
+ Many More
+ +Cost Tech Schedule
= ?
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Probability Density FunctionProbability Density Function
Cell B28 Frequency Chart
Mean = $2,198
4,978 Trials Shown
.000
.006
.012
.018
.024
0
29.2
58.5
87.7
117
$1,250 $1,750 $2,250 $2,750 $3,250
Forecast: Results=
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Cumulative Distribution FunctionsCumulative Distribution Functions
Cell B28 Cumulative Chart 4,977 Trials Shown
.000
.249
.498
.747
.995
0
4977
$1,250 $1,750 $2,250 $2,750 $3,250
Forecast: Results=
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Statistical ResultsStatistical Results
Forecast: Statistic Value
Trials 5,000
Mean $2,195
Median (approx.) $2,147
Mode (approx.) $2,100
Standard Deviation $385
Variance $148,391
Skewness 0.47
Kurtosis 2.79
Coeff. of Variability 0.18
Range Minimum $1,318
Range Maximum $3,466
Range Width $2,148
Mean Std. Error $5.45
Mean = first momentVariance = second momentStd. Deviation = (Variance)1/2
Skewness = third momentKurtosis = fourth moment
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Total Forecast Results in FY05 $M
0% $1,24210% $1,71920% $1,84230% $1,94740% $2,04350% $2,13560% $2,23170% $2,35980% $2,50790% $2,709100% $3,389
Cum
ulat
ive
Pro
babi
lity
Not
to E
xcee
d
PercentilesPercentiles
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
25
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
26
Development Schedule
Lif
e C
ycle
Cos
t
Min Optimal
Typ
ical
Cost Schedule Curve Cost Schedule Curve
Parallel EffortMore ECPLess Mature Design
Fixed CostTechnology Outdate
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
27
Defense
Schedule GoalsSchedule Goals
Development Schedule
Cos
t
Commercial Drivers• Technology Drives Schedule• Constraint Schedule• Goal is ROI Maximization
DoD Drivers• Funding Drives Schedule• Unconstrained Schedule• Goal is Cost Reduction
Commercial
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
28
DoD Program Schedule DriversDoD Program Schedule Drivers
Development Schedule
Cos
t
Cycle Time Reduction Goal
Funding Allocation
Acquisition Process
Program Execution
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
29
Development Schedule
DoD
Pre-1992 Starts 132 months (11 years)
Post-1992 Starts 89 months (7.4 years) on-going
F-22 216 months (18 years) IOC 2004
Comanche 264 months (22 years) IOC 2006
Commercial
Boeing 777 54 months (4.5 Years)
Mean Cycle Time to IOC
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
FY95 FY96 FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY04 FY05 FY06
TOTAL
ENGINE
IN-HOUSE
Example of R&D Funding Profile
FY09FY07 FY08
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
31
Historical Perspective
Program Managers’ Priority
• Superior Performance
• Low Acquisition Cost
• Low Operation Cost
• Shortened Schedule
Shortening the Cycle TimeProvides Opportunity for Cost
Reduction
78% of Programs had less than10% Schedule Slippage 92% of Programs BAFO had
Raised no Issue with theGovernment Proposed
Schedule
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
32
Reduced Cycle Time Opportunity
AS-ISTO-BE
x x/2 x/2
Program 1
Program 2Program 3
Program 4
Program 5
Program 6
Program 1
Program 2
Program 3
Program 4
Program 5
Program 6
Cycle Time Result From:
- Starting Programs too Soon
- Starting Too many Programs
- Inefficient Acquisition Process
Reduced Cycle Time will Result In:
- Fielding of Newer Technology
- Lower Cost per Project
- Reduced Exposure to Annual Funding Reduction
Cos
t
Business Case
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
33
Reasons for RiskReasons for Risk
Technical Programmatic Cost Schedule
PhysicalProperties
MaterialAvailability
Sensitivity toAssumptions
Degree ofConcurrency
Materialproperties
SkillRequirement
Sensitivity toTechnical Risk
Sensitivity toTechnical Risk
SoftwareComplexity
EnvironmentalImpact
Sensitivity toProgrammaticRisk
Sensitivity toProgrammaticRisk
IntegrationInterface
ContractorsStability
Sensitivity toSchedule Risk
Number ofCritical Path
RequirementChanges
Funding Profile EstimatingErrors
EstimatingErrors
OperationalEnvironment
PoliticalAdvocacy
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
Cost Risk Analysis ChallengesCost Risk Analysis Challenges
• Information Organization, Structure, and Mapping• Expert Identification and Communication• Interdependency of Risk Issues• Impact of Global Factors (e.g., Industrial Base)• Modeling• Probability Distribution Assumptions• Packaging Range Estimate• Interpretation of the Results• Risk Mitigation Plan• PPBES and Point Estimates
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
35
Summary
• Funding Allocation Puts more Constraint on the Program Schedules than Acquisition Process and Program Execution
• Shortening the Cycle Time Provides Cost Reduction Opportunity with Greater Operational Capability
• Cost Schedule Risk Analysis Tools and Techniques Should be Utilized to Demonstrate the Effect of Schedule Trade-off
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
36
At the end of the day, the cost analyst ends up being the system architect-integrator-engineer as
the details are worked through.
Dep
uty
Ass
ista
nt
Sec
reta
ry o
f th
e A
rmy
for
Co
st &
D
epu
ty A
ssis
tan
t S
ecre
tary
of
the
Arm
y fo
r C
ost
&
Eco
no
mic
s (D
AS
A-C
E)
Eco
no
mic
s (D
AS
A-C
E)
37
System Descriptions, Cost Models
Input Process Output ModelInput Process Output ModelInput Process Output ModelInput Process Output Model
Cost Analysis ProcessDatabases, Tools, & Models
Requirements Acquisition Assumptions
Design Parameters Risk Assessment
ASARCIT OIPTCAIG, DABAPBPPBESAOA / EA
Inputs(Descriptions)
Process(Models)
Outputs(Costs)
Types of Cost Analysis
Estimate Known Solving For: How Much Does
the System Cost?System Costs
How to Cost a System?
Historical System Description and Costs
Cost Models / CER
NEWApproach
What System Can I Get For My Dollars?
Cost TargetsSystem Descriptions, Design &
Performance Parameters
OLDApproach
Initiated Development of Performance Based and Design Based Cost Models
Models
Supports
37