developed by© 2002 scea. all rights reserved. costprof unit iii - module 91 cost risk analysis how...

Unit III - Module 9 1

developed by© 2002 SCEA. All rights reserved.

CostPROF

Cost Risk Analysis

How to adjust your estimate for historical cost growth



CostPROF

Unit IndexUnit I – Cost EstimatingUnit II – Cost Analysis TechniquesUnit III – Analytical Methods

6. Basic Data Analysis Principles7. Learning Curves8. Regression Analysis9. Cost Risk Analysis10.Probability and Statistics

Unit IV – Specialized CostingUnit V – Management Applications



CostPROF

Outline• Introduction to Risk• Model Architecture• Historical Data Analysis• Model Example• Summary• Resources



CostPROF

Introduction to Risk• Overview• Definitions• Types of Risk• Risk Process



CostPROF

Overview• Risk is a significant part of cost estimation

and is used to allow for cost growth due to anticipatable and un-anticipatable causes

• There are several approaches to risk estimation

• Incorrect treatment of risk, while better than ignoring it, creates a false sense of security

• Risk is perhaps best understood through a detailed examination of an example method



CostPROF

Definitions• Cost Growth:

– Increase in cost of a system from inception to completion

• Cost Risk:– Predicted Cost Growth.

In other words:Cost Growth = actualsCost Risk = projections

In other words:Cost Growth = actualsCost Risk = projections



CostPROF

Types of Risk• Cost Growth = Cost Estimating Growth +

Sked/Tech Growth + Requirements Growth + Threat Growth

• Cost Risk = Cost Estimating Risk + Sked/Technical Risk + Requirements Risk + Threat Risk

– Cost Estimating Risk: Risk due to cost estimating errors, and the statistical uncertainty in the estimate

– Schedule/Technical Risk: Risk due to inability to conquer problems posed by the intended design in the current CARD or System Specifications

– Requirements Risk: Risk resulting from an as-yet-unseen design shift from the current CARD or System Specifications arising due to shortfalls in the documents

• Due to the inability of the intended design to perform the (unchanged) intended mission

• We didn’t understand the solution– Threat Risk: Risk due to an unrevealed threat; e.g. shift from the

current STAR or threat assessment• The problem changedO

ften

im

plicit

or

om

itte

d

1 2



CostPROFBasic Flow of the Risk Process

Structure & ExecutionIncludes the organization,

the mathematical assumptions,and how the model runs

Structure & ExecutionIncludes the organization,

the mathematical assumptions,and how the model runs

Inputs Outputs

From the cost analyst and technical experts• The CARD• Expert rating/scoring• Point Estimate

To the decision maker and the cost analyst• Means• Standard Deviations• Risk by CWBSInputs and outputs, although outside the

purview of the risk analyst, are determined by the structure and

execution of the risk model

Inputs and outputs, although outside the purview of the risk analyst, are

determined by the structure and execution of the risk model



CostPROF

Engineers• Work in physical

materials, with – Physics-based responses– Physical connections

• Typically examine or discuss a specific outcome– System Parameters– Designs

• Typically seek to know:– Given this solution, what

will go wrong?– Are my design margins

enough?

Engineers’ and Cost Analysts’ View of Risk

Cost Analysts• Work in dollars and

parameters, with– Statistical relationships– Correlation

• Typically examine or discuss a general outcome set– Probability distribution– Statistical parameters such

as mean and standard deviation• Typically seek to know:

– Given this relationship, what is the range of possibilities?– Are my cost margins enough?



CostPROF

Model Architecture• Inputs• Structure• Execution



CostPROFGeneral Model Architecture

– Coverage & Partition• Cost Estimating• Schedule / Technical• Requirements• Threat

– Assigning Cost to Risk• CERs• Direct Assessment of

Distribution Parameters • Factors • Rates

– Below-the-Line• Yes• No

– Distribution• Normal • Log Normal • Triangular • Beta• Bernoulli

– Correlation• Functional • Relational • Injected• None

Scori

ng

Org

an

izati

on

• Interval w/ objective criteria• Interval• Ordinal• None

• Monte Carlo• Method of Moments• Deterministic

Pro

bab

ilit

y M

od

el

Cro

ss

Ch

ecks

• Means• CVs• Inputs

Tip: Higher is better except

in Cross Checks

• Historical• Domain Experts• Conceptual

Dollar

Basis

Inputs

Com

pu

-ta

tion

Structure

Execution



CostPROF

SCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTSSCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTS

Inputs – Scoring• Interval with objective criteria

– Set scoring based on objective criteria, and for which the distance (interval) between scores has meaning. (Note: the below example is also Ratio, because it passes through the origin.)

• A schedule slip of 1 week gets a score of 1, a slip of 2 weeks gets a score of 2, a slip of 4 weeks gets a 4, a slip of 5 weeks gets a score of 5, etc.

• The difference between a score of 1 and 2 is as big as a difference between score of 4 and 5

• A scale is interval if it acts interval under examination*

“Nominal, ordinal, interval, and ratio typologies are misleading,” P.F. Velleman and L. Wilkinson, The American Statistician, 1993, 47(1), 65-72

8



CostPROF

Inputs – Scoring• Interval

– Set scoring for which the distance (interval) between scores has meaning

• Low risk is assigned a 1, medium risk is assigned a 5, and a high risk is assigned a 10

• Note that it is not immediately clear that the scale is interval, but it is surely not subject to objective criteria.

• Ordinal– Score is relative to the measurement

• e.g., difficulty in achieving schedule is high, medium, or low

• None



CostPROF

SCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTSSCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTS

Inputs – Dollar Basis• Historical

– Actual costs of similar programs or components of programs are used to predict costs

• Domain Experts– Persons with expertise regarding similar

programs or program components assess the cost based on their experience

• Conceptual– An arbitrary impact is assigned

• Any scale without a historical basis or expert assessment is conceptual



CostPROF

SCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTSSCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTS

Org – Coverage & Partition• How the four types of risk are

covered and partitioned– Cost Estimating– Schedule/Technical– Requirements– Threat

These risk types may be covered implicitly or explicitly in any

combination.

These risk types may be covered implicitly or explicitly in any

combination.



CostPROF

SCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTSSCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTS

Org – Assigning Cost to Risk

• Risk CERs: Equations are developed that reflect the relationship between an interval risk score and the cost impact of the risk (this might also be termed a Risk Estimating Relationship (RER))– These equations amount to the same thing as CERs

used in the cost estimate– e.g., Risk Amount = 0.12 * Risk Score

• Direct Assessment of Distribution Parameters: Costs are captured in shifts of parameters of the risk, e.g., shifted end points for triangulars, shifted end points or means for betas, etc.– Note: Scoring is completely eliminated from this

mapping method– e.g., triangles assessed by domain experts

9



CostPROF

SCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTSSCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTS

Org – Assigning Cost to Risk

• Factors: Fractions or percents are used in conjunction with the scores and the cost of the component or program – e.g., a score of 2 increases the cost of the component

by 8%– Antenna Risk Score = 2– Cost of Antenna = $4090K– Risk Amount = 0.08 * 4090K = $327.2K

• Rates: Predetermined costs are associated with the scores – e.g., a score of 2 has a cost of $100K– Antenna Risk Score = 2– Cost of Antenna = $4090K– Risk Amount = $100K

Warning: Rates are independent of the

element’s cost.



CostPROF

SCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTSSCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTS

Org – Below-the-Line• Below-the-Line Elements

– Elements that are driven by hardware, software, and the like

– Below-the-Line Elements include:• Systems Engineering/Program Management

(SE/PM)• System Test and Evaluation (ST&E)

– Not all models account for this cost growth – Functional Correlation is another approach

to address the risk in these elements9



CostPROF

SCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTSSCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTS

Probability Model – Distribution

• Normal – Best behavior, most

iconic – Theoretically

(although not practically) allows negative costs, which spook some users

– Symmetric, needs mean shift to reflect propensity for positive growth

• Lognormal– A natural result in

non-linear CERs– Indistinguishable

from Normal at CVs below 25%

– Skewed

10

4



CostPROFProbability Model – Distribution

• Triangular – Most common– Easy to use, easy to understand– Modes, medians do not add– Skewed

• Beta– Rare now, but formerly popular– Solves negative cost and duration issues– Many parameters – simplifications like

PERT Beta are possible– Skewed

• Bernoulli– Probability is only assigned to two

possible outcomes, success and failure (p and 1-p)

– Simplest of all discrete distributions– Mean = p– Variance = p*(1-p)

10



CostPROF

SCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTSSCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTS

Probability Model – Correlation

• Functional: Arises between source and derivative variables as a result of functional dependency. The lines of the Monte Carlo are cell-referenced wherever relationships are known. – CERs are entered as equations– Cell references are left in the spreadsheet– When the Monte Carlo runs, input variables

fluctuate, and outputs of CERs reflect this

Correlation is a measure of the relation between two or more variables/WBS

elements

Correlation is a measure of the relation between two or more variables/WBS

elements

An Overview of Correlation and Functional Dependencies in Cost Risk and Uncertainty Analysis, R. L. Coleman and S. S. Gupta, DoDCAS, 1994

3



CostPROF

200

250

300

350

400

1000 1200 1400 1600 1800 2000

Recurring Production

SEP

M

200

250

300

350

400

1000 1200 1400 1600 1800 2000

Recurring Production

SE

PM

Functional Correlation• Old: No Functional

Correlation; Simulation run with WBS items entered as values

• New: Simulation run with functional dependencies entered as they are in cost model

CorrelatedCorrelatedNot CorrelatedNot Correlated

Note shift of mean, and increased variability



CostPROFProbability Model – Correlation

• Relational: Introduces the geometry of correlation and provides a substantial improvement over injected correlations, and fills a gap in FC– Relational Correlation provides insight into

• the tilt of the data, i.e. the regression line, • and the variance around the regression line

Relational Correlation: What to do when Functional Correlation is Impossible, R. L. Coleman, J. R. Summerville, M. E. Dameron, C. L. Pullen, S. S. Gupta, ISPA/SCEA Joint International Conference,2001



CostPROFProbability Model - Correlation

• Injected: Imposed by setting the correlation directly between variables without having a functional relationship.

• None: No relationship exists among the variables. The lines of the Monte Carlo are self contained.



CostPROFShortcomings of Injected Correlation

• Correlations are very hard to estimate• No check of the functional implications of

the correlations is done– This is troublesome because of the regression

line that arises when we insert a correlation. – Simply injecting arbitrary correlations of 0.2 -

0.3 to achieve dispersion is unsatisfactory as well.

• Unless the injected correlations are among elements that are actually correlated

• If correlations are actually known, no harm is done.



CostPROF

SCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTSSCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTS

Execution – Computation• Monte Carlo: A widely accepted method, used on a

broad range of risk assessments for many years. It produces cost distributions. The cost distributions give decision makers insight into the range of possible costs and their associated probabilities.

• Method of Moments: The mean and standard deviation of lower-level WBS lines are known, and are rolled up assuming independence to provide higher-level distributions. – Only provides an analysis of distribution at a top level– Easy to calculate– Negated by the rapid advances in microcomputer technology– Only works for independent elements, unless covariances are

allowed for, which is difficult.

• Deterministic: Only point values are used. No shifts or other probabilistic effects are taken into account.

10



CostPROFRisk Assessment Techniques

• Add a Risk Factor/Percentage (Minutes)– Low accuracy, no intervals

• Bottom Line Monte Carlo/Bottom Line Range/Method of Moments (Hours)– Moderate accuracy, provides intervals

• Historically based Detailed Monte Carlo (Months of non-recurring work, but recurring in days)– Time consuming non-recurring work, but with recurring

implementation being easier, accurate if done right. Provides intervals.

• Expert Opinion-Based Probability and Consequence (Pf*Cf) or Expert Opinion-Based Detailed Monte Carlo (Months)– Time consuming with no gains in recurring effort, but accurate

if done right. Provides intervals.• Detailed Network and Risk Assessment (Month)

– Time consuming with no gains in recurring effort, but accurate if done right. Provides intervals.



CostPROF

SCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTSSCORING

COMPU-TATION

ORGANI-ZATION

CROSSCHECKS

PROB.MODEL

INPUTS

Execution – Cross Checks• Means: The mean cost growth factor for WBS

items can be compared to history as a way to cross check results

• CVs: The CV of the cost growth factors for WBS items can be compared to history as a way to cross check results

• Inputs: Checks are performed on inputs or other parameters to see if historical values are in line with program assumptions– Example: Historical risk scores can be compared to

program risk scores to see if risk assessors are being realistic, and to see if the underlying database is representative of the program.

11



CostPROF

Historical Data Analysis• SARs• Contract Data• Common

Problems



CostPROF

Intro to SARs – SampleA SAR report is submitted for each year of a

program’s Acquisition cycle. The most recent SAR is used to determine cost

growth

To calculate the CGF, adjust the current estimate for quantity changes, then divide by the baseline estimate

To calculate the CGF, adjust the current estimate for quantity changes, then divide by the baseline estimate

12

Sample Program: XXX, December 31, 19XX



CostPROF

NAVAIR Cost Growth Study: A Cohorted Study of The Effects of Era, Size, Acquisition Phase, Phase Correlation and Cost Drivers , R. L. Coleman, J. R. Summerville, M. E. Dameron, C. L. Pullen, D. M. Snead, DoDCAS, 2001 and ISPA/SCEA International Conference, 2001

Contract Data• Hard to use – problems with changing baselines,

lack of reasons for variances, and access to data• Preliminary comparative analysis suggests

Contract Data mimics patterns in SAR data– Shape of distribution– Trends in tolerance for cost growth

• K-S tests find no statistically significant difference between Contract data and SAR data for programs <$1B in RDT&E– Failed to reject the null hypothesis of identical

distributions• Descriptive statistics indicate amount of Contract

Data growth and dispersion is more extreme than previously found in SAR studies

• SAR data remains the best choice for analysis and predictive modeling



CostPROF

0.0

10.0

20.0

30.0

40.0

50.0

$0 $100 $200 $300 $400 $500 $600

Contract

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

$0 $1,000 $2,000 $3,000 $4,000 $5,000 $6,000 $7,000

RAND93

CGF vs init est -- NAVAIR Contract AND RAND 93

(RDT&E) ZOOM IN w/COMMON SCALE

0.0

2.0

4.0

6.0

8.0

10.0

$0 $200 $400 $600 $800 $1,000

Contract Data RAND93

Contract Data Exploratory Analysis

Contract Data blends well: Continues trend that tolerance for growth increases as program size decreases

Contract Data blends well: Continues trend that tolerance for growth increases as program size decreases

SAR Data

Contract Data

CGF vs IPE-Contract and SAR (RDT&E)

ZOOM IN with common Scale



CostPROF

Common Problems• Most historically-based methods rely on

SARs– Adjusting for quantity – important to remove

quantity changes from cost growth– Beginning points – the richest data source is

found by beginning with EMD– Cohorting must be introduced to avoid

distortions

• EVM data is also potentially useable, but re-baselined programs are a severe complication.

• “Applicability” and “currency” are the most common criticisms

15



CostPROF

Applicability and Currency• Applicability: “Why did you include

that in your database?”– Virtually all studies of risk have failed

to find a difference among platforms (some exceptions)

– If there is no discoverable platform effect, more data is better

• Currency: “But your data is so old!”– Previous studies have found that post-

1986 data is preferable– Data accumulation is expensive



CostPROF

Model Example• Overview• Scoring• Database• Sample Outputs



CostPROFExample Model Architecture

– Coverage & Partition• Cost Estimating• Schedule / Technical• Requirements• Threat

– Assigning Cost to Risk• CERs• Direct Assessment of

Distribution Parameters • Factors • Rates

– Below-the-Line• Yes• No

– Distribution• Normal • Log Normal • Triangular • Beta• Bernoulli

– Correlation• Functional • Relational• Injected• None

Scori

ng

Org

an

izati

on

• Interval w/ objective criteria• Interval• Ordinal• None

• Monte Carlo• Method of Moments• Deterministic

Pro

bab

ilit

y M

od

el

Cro

ss

Ch

ecks

• Means• CVs• Inputs

• Historical• Domain Experts• Conceptual

Dollar

Basis

Inputs

Com

pu

-ta

tion

Structure

ExecutionTip: Higher is better except

in cross checks

13



CostPROF

Assessment Approach

• Develop a cost estimating risk distribution for each CWBS element

• Develop a schedule/technical risk distribution for each WBS entry for: – Hardware– Software– Note that “Below-the-line” WBS elements get

risk from Above-the-line WBS elements via Functional Correlation

• Combine these risk distributions and the point estimate using a Monte Carlo simulation



CostPROF

Example Model in Blocks

FunctionalCorrelation

StandardErrors &

SEEs

MappingRisk

ScoringMonteCarlo

Risk Report

Risk Report

CARDCARD

IPEIPE

Cost Estimating Risk

Sked/Tech Risk

Cost Risk Analysis of the Ballistic Missile Defense (BMD) System, An Overview of New Initiatives Included in the BMDO Risk Methodology, R. L. Coleman, J. R. Summerville, D. M. Snead, S. S. Gupta, G. E. Hartigan, N. L. St. Louis, DoDCAS, 1998 (Outstanding Contributed Paper), and ISPA/SCEA International Conference, 1998



CostPROFCost Estimating Risk Assessment

• Consists of a standard deviation and a bias associated with the costing methodologies– Standard deviation comes from the

CERs and factors– Bias is a correction for

underestimating

14



CostPROFSked/Tech Risk Assessment

• Technical risk is decomposed into categories and each category into sub categories– Hardware sub categories:

• Technology Advancement, Engineering Development, Reliability, Producibility, Alternative Item and Schedule

– Software sub categories:• Technology Approach, Design

Engineering, Coding, Integrated Software, Testing, Alternatives, and Schedule



CostPROFHardware Risk Scoring Matrix

Risk Risk Scores (0=Low, 5=Medium, 10=High)Categories 0 1-2 3-5 6-8 9-10

1Technology

AdvancementCompleted (State

of the Art)

Minimum Advancement

Required

Modest Advancement

Required

Significant Advancement

RequiredNew Technology

2 Engineering Development

Completed (Fully Tested)

PrototypeHW/SW

DevelopmentDetailed Design Concept Defined

3Reliability

Historically High for Same Item

Historically High on Similar Items

Known Modest Problems

Known Serious Problems

Unknown

4Producibility

Production & Yield Shown on

Same Item

Production & Yield Shown on

Similar Items

Production & Yield Feasible

Production Feasible & Yield

Problems

No Known Production Experience

5

Alternate Item

Exists or Availability on

Other Items Not Important

Exists or Availability of

Other Items Somewhat Important

Potential Alternative Under

Development

Potential Alternative in

Design

Alternative Does Not Exist & is

Required

6Schedule Easily Achievable Achievable

Somewhat Challenging

Challenging Very Challenging



CostPROFCalculating Sked/Tech Risk Endpoints

• Technical experts score each of the categories from 0 (no risk) to 10 (high risk)

• Each category is weighted depending on the relevancy of the category– Weights are allowed, but rarely used

• Weighted average risk scores are mapped to a cost growth distribution– This distribution is based on a database of cost

growth factors of major weapon systems collected from SARs. These programs range from those which experienced tremendous cost growth due to technical problems to those which were well managed and under budget.



CostPROF

Sked/Tech Score Mapping

Typical Risk Assessment Score Mapped to Factor--RDT&E

0

0.5

1

1.5

2

2.5

3

3.5

0 1 2 3 4 5 6 7 8 9 10

Risk Score

Dis

trib

uti

on

En

d P

oin

ts

Maximum possible cost growth

Average cost growth

Minimum possible cost growth

MIN=0.77

AVG=1.17

MAX=1.58

MIN=0.61

AVG=1.28

MAX=1.96MIN=0.46

AVG=1.40

MAX=2.34



CostPROFSked/Tech Risk Distribution

Bars are the frequency

of occurrence of each

risk score

Model

1 3 5 7 9 10

This is the compositePDF for all SARs

S/T Risk Score

These are the PDFs for 3 risk scores above. More risk has higher mode, wider base, all are symmetric.

MIN=0.77

AVG=1.17

MAX=1.58MIN=0.61

AVG=1.28

MAX=1.96MIN=0.46

AVG=1.40

MAX=2.34



CostPROF

Cost Growth Database

24

15

28

20

89

4

7

1 13

5

-30% -20% -10% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 400%

Cost Decrease Cost Increase

No Change

Risk appears skewed, perhaps Triangular or Lognormal

Risk appears skewed, perhaps Triangular or Lognormal

This distribution, found in databases, is the result of a blending of a family of distributions as shown on the previous slide.

This distribution, found in databases, is the result of a blending of a family of distributions as shown on the previous slide.



CostPROFRisk Report Sample Output

Statistics: ValueTrials 5000Mean 172.12Median 172.33Mode ---Standard Deviation 43.32Variance 1877.03Skewness 0.01Kurtosis 2.88Coeff. of Variability 0.25Range Minimum 22.38Range Maximum 318.94Range Width 296.56Mean Std. Error 0.61

Forecast: DD 21$

Crystal Ball Report

Simulation stopped on 1/3/01 at 18:24:47Simulation started on 1/3/01 at 18:24:18

Frequency Chart

.000

.006

.012

.019

.025

0

31

62

93.00

124

50.00 112.50 175.00 237.50 300.00

5,000 Trials 13 Outliers

Forecast: DD 21$ 5



CostPROF

Example Cost Estimate with Risk – R&D

0

50

100

150

Initial PointEstimate

Add CostEstimating

Risk

AddSched/Tech

Risk

S/T Risk

CE Risk

Init Pt Est

$

8.7%25%

33.7%

6

7

Note: These are means – there is an associated confidence interval, not

portrayed.

Note: These are means – there is an associated confidence interval, not

portrayed.



CostPROF

Summary– Why include risk?

• Risk adjusts the cost estimate so that it more closely represents what historical data and experts know to be true … it predicts cost growth

– How to treat risk?• We have seen an overview of the many different

options in terms of inputs, the structure of the risk model, and how to execute the risk model

• The choices are varied, but it is important that the model “fits together” and that it predicts well.

– Closing thought: Always include cross checks to support the accuracy of the model and the specific results for a program

• The model may seem right, but will it (did it) predict accurate results?



CostPROF

Risk Resources – Books•Against the Gods: The Remarkable Story of Risk, Peter L. Bernstein, August 31, 1998, John Wiley & Sons•Living Dangerously! Navigating the Risks of Everyday Life, John F. Ross, 1999, Perseus Publishing•Probability Methods for Cost Uncertainty Analysis: A Systems Engineering Perspective, Paul Garvey, 2000, Marcel Dekker•Introduction to Simulation and Risk Analysis, James R. Evan, David Louis Olson, James R. Evans, 1998, Prentice Hall•Risk Analysis: A Quantitative Guide, David Vose, 2000, John Wiley & Sons



CostPROF

Risk Resources – Web• Decisioneering

– Makers of Crystal Ball for Monte Carlo simulation

– http://www.decisioneering.com

• Palisade– Makers of @Risk for Monte Carlo

simulation– http://www.palisade.com



CostPROF

Risk Resources – Papers•Approximating the Probability Distribution of Total System Cost, Paul Garvey, DoDCAS 1999 • Why Cost Analysts should use Pearson Correlation, rather than Rank Correlation, Paul Garvey, DoDCAS 1999 • Why Correlation Matters in Cost Estimating , Stephen Book, DoDCAS 1999•General-Error Regression in Deriving Cost-Estimating Relationships, Stephen A. Book and Mr. Philip H. Young, DoDCAS 1998 • Specifying Probability Distributions From Partial Information on their Ranges of Values, Paul R. Garvey, DoDCAS 1998 •Don't Sum EVM WBS Element Estimates at Completion, Stephen Book, Joint ISPA/SCEA 2001•Only Numbers in the Interval –1.0000 to +0.9314… Can Be Values of the Correlation Between Oppositely-Skewed Right-Triangular Distributions, Stephen Book , Joint ISPA/SCEA 1999



CostPROF

Risk Resources – Papers•An Overview of Correlation and Functional Dependencies in Cost Risk and Uncertainty Analysis, R. L. Coleman, S. S. Gupta, DoDCAS, 1994•Weapon System Cost Growth As a Function of Maturity, K. J. Allison, R. L. Coleman, DoDCAS 1996•Cost Risk Estimates Incorporating Functional Correlation, Acquisition Phase Relationships, and Realized Risk, R. L. Coleman, S. S. Gupta, J. R. Summerville, G. E. Hartigan, SCEA National Conference, 1997•Cost Risk Analysis of the Ballistic Missile Defense (BMD) System, An Overview of New Initiatives Included in the BMDO Risk Methodology, R. L. Coleman, J. R. Summerville, D. M. Snead, S. S. Gupta, G. E. Hartigan, N. L. St. Louis, DoDCAS, 1998 (Outstanding Contributed Paper) and ISPA/SCEA International Conference, 1998



CostPROF

Risk Resources – Papers•Risk Analysis of a Major Government Information Production System, Expert-Opinion-Based Software Cost Risk Analysis Methodology, N. L. St. Louis, F. K. Blackburn, R. L. Coleman, DoDCAS, 1998 (Outstanding Contributed Paper), and ISPA/SCEA International Conference, 1998 (Overall Best Paper Award)•Analysis and Implementation of Cost Estimating Risk in the Ballistic Missile Defense Organization (BMDO) Risk Model, A Study of Distribution, J. R. Summerville, H. F. Chelson, R. L. Coleman, D. M. Snead, Joint ISPA/SCEA International Conference 1999•Risk in Cost Estimating - General Introduction & The BMDO Approach, R. L. Coleman, J. R. Summerville, M. DuBois, B. Myers, DoDCAS, 2000•Cost Risk in Operations and Support Estimates, J. R. Summerville, R. L. Coleman, M. E. Dameron, SCEA National Conference, 2000



CostPROF

Risk Resources – Papers•Cost Risk in a System of Systems, R.L. Coleman, J.R. Summerville, V. Reisenleiter, D. M. Snead, M. E. Dameron, J. A. Mentecki, L. M. Naef, SCEA National Conference, 2000• NAVAIR Cost Growth Study: A Cohorted Study of the Effects of Era, Size, Acquisition Phase, Phase Correlation and Cost Drivers, R. L. Coleman, J. R. Summerville, M. E. Dameron, C. L. Pullen, D. M. Snead, ISPA/SCEA Joint International Conference, 2001•Probability Distributions of Work Breakdown Structures,, R. L. Coleman, J. R. Summerville, M. E. Dameron, N. L. St. Louis, ISPA/SCEA Joint International Conference, 2001•Relational Correlation: What to do when Functional Correlation is Impossible, R. L. Coleman, J. R. Summerville, M. E. Dameron, C. L. Pullen, S. S. Gupta, ISPA/SCEA Joint International Conference,2001•The Relationship Between Cost Growth and Schedule Growth, R. L. Coleman, J. R. Summerville, DoDCAS, 2002•The Manual for Intelligence Community CAIG Independent Cost Risk Estimates, R. L. Coleman, J. R. Summerville, S. S. Gupta, DoDCAS, 2002



CostPROF

Advanced Topics• Relational Correlation and the

Geometry of Regression



CostPROFGeometry of Bivariate Normal Random Variables

(μx, μy)

σx

σyρ=.75

μx

σy

σx

μy

x

y

ρ=0

The dispersion and axis tilt of the “data cloud” is a function of

correlation:• less correlation, more dispersion about the axis

• more correlation, more axis tilt

tilt



CostPROF

This line is with perfect correlation …

The slope that would be true if ρ = 1

Implications for Regression Line

(μx, μy)

σx

σy

2σx ρ=.75

b= μy- ρσy / σx * μxb

μx

σy

σx

μy

x

y

2σy

This line has correlation

added

y = ρ(σy / σx) (x- μx) + μy



CostPROF

Intercept b varies with ρ, σx, σy, μx, and μy

Geometry of Regression Line

(μx, μy)

σx

σy

2σx

ρ=1

ρ=-1Dispersion

varies with ρb= μy- ρσy / σx * μx

b

μx

σy

σx

Slope m varies with ρ, σx, σy

μy

x

y

ρ=02σy

Ran

ge o

f in

terc

epts

Range of slopes

y = ρ(σy / σx) (x- μx) + μy

The regression line of y on x depends on their means, their standard deviations and their

correlation

developed by© 2002 scea. all rights reserved. costprof unit iii - module 91 cost risk analysis how...

Documents

cost risk analysis

actuals cost risk

completion cost risk

risk analyst

types of risk cost growth

overview risk

cost analyst

cost estimation