patterson.dan

Using the Risk Register in Integrated Cost/Schedule Risk Analysis with Monte Carlo Simulation

David T. Hulett, Ph.D.Hulett & Associates, LLC

Dr. Dan Patterson, PMPAcumen

NASA PM ChallengeDaytona Beach, FL

February 24-25, 2009

© 2009 Hulett & Associates, AcumenPM


Agenda

• Explain “Risk Factors” approach• Apply Risk Factors to schedule and cost risk• Apply Risk Factors to simple space vehicle

development schedule as an example


Limitations with the Traditional 3-point Estimate of Activity Duration

• Typical schedule risk analysis starts with the activity that is impacted by risks– Estimates the 3-point estimate for optimistic, most

likely and pessimistic duration– Implies the risk is 100% likely with uncertain impact

• Which risks cause the most overall schedule risk? Cannot say directly, but indirectly:– Sensitivity to activity durations– Criticality of activity durations


Some Problems with Traditional Approach

• Can tell which activities are crucial, but not directly which risks are driving

• Makes poor use of the Risk Register that is usually available

• Cannot decompose the overall schedule risk into its components BY RISK– Ability to assign the risk to its specific risk

drivers helps with communication of risk causes and risk mitigation


We Propose a Different Approach: Start with the Risks Themselves

• Drive the schedule risk by the risks already prioritized in the Risk Register

• For each risk, specify:– Probability it will occur (NEW)– Impact on time if it does (multiplicative factors, allows it to

affect different duration activities)– Assign to the activities it will affect

• Starting with the risks themselves gives us benefits– Links qualitative analysis to the quantitative analysis– Estimates the impact of specific risks for prioritized

mitigation purposes– Correlations between activities modeled automatically


Simple Example of Risk Register Risks

• Use the Risk Factors module in Pertmaster 8• Collect probability and impact data on risks• Map the risks to activities


Risk Factors Mechanics (1)

• The risk factor is assigned to one or several activities, affecting their durations by a multiplicative factor– E.g., the factor may be .90 for optimistic, 1.0 for most

likely and 1.25 for pessimistic– These factors multiply the schedule durations of the

activities to which they are assigned • Risks can be assigned to one or more activities• Activity durations can be influenced by one or

more risks


Risk Factors Mechanics (2)

• Risk Factors are assigned a probability of occurring on any iteration – When the risk occurs, the factor used is chosen at

random from the 3-point estimate and operates on all activities to which it is assigned

– When not occurring on an iteration the risk factor takes the value 1.0, a neutral value

• When an activity is influenced by more than one risk, their factors are multiplied together, if they happen, on any iteration


Risk Factor Probability is 100%, Factor can be + or -

90 95 100 105 110 115

Distribution (sta rt o f in te rva l)

0

50

100

150

200

250

Hits

0% 90

5% 94

10% 95

15% 96

20% 97

25% 98

30% 99

35% 99

40% 100

45% 101

50% 101

55% 102

60% 103

65% 104

70% 104

75% 105

80% 106

85% 108

90% 109

95% 111

100% 115

Cum

ulat

ive

Freq

uenc

y

0 01 0 - C o n s tru c tio n : D u ra tio n

Here the Ranges are based on deviations + and – from the Plan. Probability is 100%

For the examples we use an activity with 100 days in the schedule


Assigning a Probability Less than 100%

100 110 120 130

Distribution (start of interval)

0

200

400

600

800

1000

1200

Hits

0% 100

5% 100

10% 100

15% 100

20% 100

25% 100

30% 100

35% 100

40% 100

45% 105

50% 107

55% 109

60% 110

65% 111

70% 113

75% 114

80% 116

85% 118

90% 120

95% 123

100% 130

Cum

ulat

ive

Freq

uenc

y

0040 - Technology Design : Duration

95 100 105 110 115


0

500

1000

1500

2000

Hits

0% 91

5% 97

10% 99

15% 100

20% 100

25% 100

30% 100

35% 100

40% 100

45% 100

50% 100

55% 100

60% 100

65% 100

70% 100

75% 100

80% 100

85% 101

90% 103

95% 107

100% 115

Cum

ulat

ive

Freq

uenc

y

0010 - Construction : Duration

Spike contains 70% of the probability

Spike contains 40% of the probability


Assigning More than One Risk to an Activity

• If more than one risk is acting on an activity, the resulting ranges are the multiplication of the factors

• The activity duration range is derived from the risk factors that affect the risk– Model how the activity duration range is

generated– Focus on the causes of activity duration

ranges


Two Risks affect One Activity using Factors that Occur 100%

100 110 120 130 140


0

20

40

60

80

100

120

140

Hits

0% 93

5% 102

10% 104

15% 106

20% 108

25% 109

30% 110

35% 111

40% 113

45% 114

50% 115

55% 116

60% 117

65% 118

70% 119

75% 121

80% 123

85% 124

90% 127

95% 130

100% 144

Cum

ulat

ive

Freq

uenc

y


Range from 90 to 150 days, Peak about 113 days


Two Risks with Less than 100% Probability Affecting one Activity

100 110 120 130 140


0

100

200

300

400

500

600

700

800

900

1000

1100

Hits

0% 91

5% 97

10% 99

15% 100

20% 100

25% 100

30% 100

35% 100

40% 100

45% 101

50% 102

55% 104

60% 106

65% 108

70% 110

75% 111

80% 113

85% 116

90% 119

95% 123

100% 144

Cum

ulat

ive

Freq

uenc

y


The spike at 100 days represents (1) the likelihood that neither risk occurs and (2) the chance that 100 days is picked when one or both occur

Risk Factors Model How Correlation OccursCoefficients are Calculated (1)


Risk #1P = 50%, Factors

.95, 1.05, 1.15

Activity A Activity B

Activities A and B Correlation is Calculated to be 100%

Risk Factors Model How Correlation OccursCoefficients are Calculated (2)



.95, 1.05, 1.15

Activity A Activity B

Activities A and B Correlation is Calculated to be 48%


.8, .95, 1.05


1.0, 1.1, 1.2

Correlation is modeled as it is caused in the projectCorrelation coefficients are generated, not guessed


Sensitivity to the Risk Factors

The tornado diagram focuses on risks, not activities


Simple 2-Stage Space Vehicle Schedule

Software used: Pertmaster v. 8


Case Study: Simple Space Vehicle Development Schedule

• 87 month schedule• 11 work activities linked, 3 major milestones• Beginning 3 March 2008• PDR on 11 September 2009• CDR on 3 June 2011• Delivery to launch site 12 June 2015


Two Types of Risk

• Background risk based on typical general risk, estimating error– Used Quick Risk of -5% and +10%

•• Discrete risksDiscrete risks derived from Risk Register– Summarized from detailed Risk Register– These have a probability of occurring and an

impact on specific activities if they do– Parallel to their Risk Register information– Risk Register is used in data collection


Standard 3-point Range Representing Schedule Estimating Error

Background risk:

Optimistic -5% Pessimistic +10%


Results with Schedule Estimating Error Only

Deterministic: 12JUN15 is <1%

P-80 is 30SEP15, about 3.5 monthslater than planned

Spread from P-5 to P-95 is 5JUL15 to 27OCT15 for 3.7 months

31/May/15 08/Sep/15 17/Dec/15


0

100

200

300

400

Hits

0% 14/May/15

5% 06/Jul/15 10% 17/Jul/15

15% 24/Jul/15

20% 31/Jul/15 25% 07/Aug/15

30% 12/Aug/15

35% 18/Aug/15 40% 21/Aug/15

45% 26/Aug/15 50% 01/Sep/15

55% 04/Sep/15

60% 09/Sep/15 65% 14/Sep/15

70% 18/Sep/15

75% 24/Sep/15 80% 30/Sep/15

85% 07/Oct/15

90% 15/Oct/15 95% 27/Oct/15

100% 30/Dec/15

Cum

ulat

ive

Freq

uenc

y

Spacecraft ProgramEntire Plan : Finish Date

Risk Analysis on Space Vehicle ProjectRisk Factors are from Risk Register


Time Impact Range Cost Impact Range

Probability Minimum Most Likely Maximum Minimum Most Likely Maximum

Requirements have not been decided

30% 95% 105% 120%

Several alternative designs considered

60% 95% 100% 115%

New designs not yet proven 40% 96% 103% 112%

Fabricaton requires new materials

50% 96% 105% 115%

Lost know‐how since last full spacecraft

30% 95% 100% 105%

Funding from Congress is problematic

70% 90% 105% 115%

Schedule for testing is aggressive

100% 100% 120% 130%

Cost Risk is based on immature data

100% 95% 105% 110%

Mapping Risks to Activities (1)


Risk Requirements Definition

FS Preliminary Design

FS Final Design FS Fabrication

Test FS Engine

Requirements Not Complete X

Alternative Designs Possible X

Designs Not Proven XNew Materials in Fabrication X

Lost Know-How XFunding Problematic X X X XTesting Schedule Aggressive X

Cost Estimate is based on Immature Data

X X X X X

Mapping Risks to Activities (2)


RiskUS Preliminary Design

US Final Design US Fabrication US Test Integration Integratio

n Testing

Requirements Not CompleteAlternative Designs Possible X

Designs Not Proven X

New Materials in Fabrication X

Lost Know-How X XFunding Problematic X X X X X X

Testing Schedule Aggressive X X

Cost Estimate is based on Immature Data

X X X X X X


Results Adding Risk Factors to the Background Risk

Baseline 12JUN 15 is only 3% likely

The 80th percentile (P-80) is 26MAY16, 11.5 months later

Spread P-5 to P-95 is 13JUL15 to 5OCT16, for 15.5 months

20/Feb/15 08/Sep/15 26/Mar/16 12/Oct/16 30/Apr/17


0

50

100

150

200

250

300

350

Hits

0% 21/Jan/15

5% 13/Jul/15

10% 27/Aug/15

15% 25/Sep/15

20% 15/Oct/15

25% 02/Nov/15

30% 16/Nov/15

35% 01/Dec/15

40% 16/Dec/15

45% 30/Dec/15

50% 15/Jan/16

55% 04/Feb/16

60% 24/Feb/16

65% 15/Mar/16

70% 06/Apr/16

75% 02/May/16

80% 26/May/16

85% 01/Jul/16

90% 11/Aug/16

95% 05/Oct/16

100% 03/May/17

Cum

ulat

ive

Freq

uenc

y

Spacecraft for PMChallenge 2009Entire Plan : Finish Date


Activity Tornado Chart from All-In Simulation

Risky Activities:Fabrication, Integration, Final Design, Preliminary Design, TestingThis is the typical sensitivity analysis of 3-point estimating

61%

62%

63%

69%

69%

76%

76%

78%

80%

81%00025 - US Fabrication

00011 - FS Fabrication

00028 - Integration

00009 - FS Final Design

00023 - US Final Design

00021 - US Preliminary Design

00007 - FS Preliminary Design

00012 - Test FS Engine

00029 - Integration Testing

00026 - US Test

Spacecraft for PMChallenge 2009Duration Sensitivity


Risk Factor Tornado from All-In Simulation

The main RISK, however, is funding from Congress, which affects all activities. This is the main risk to mitigate, if possible

6 - Funding from Congress is problematic

4 - Fabricaton requires new materials

3 - New designs not yet proven

7 - Schedule for testing is aggressive

5 - Lost know-how since last full spacecraft

2 - Several alternative designs considered

1 - Requirements have not been decided

0.00 0.20 0.40 0.60 0.80

Correlation

Driving Schedule Risk Factors

Contribution of Each Risk to the Time Contingency (1)


Explain the Contingency to the P-80

P-80 Date Contribution of Risk

All Risks In 26-May-16 Days Saved % of Contingency

Specific Risks Taken Out in Order

Funding Risk 22-Jan-16 125 36%

Testing Schedule is Aggressive 1-Dec-15 52 15%

Design Requires New Materials 28-Oct-15 34 10%

New Design Risk 15-Oct-15 13 4%

Alternative Design Risk 6-Oct-15 9 3%

Requirements Risk 1-Oct-15 5 1%Lost Know How Risk 30-Sep-15 1 0%

Background Schedule Estimating Risks

Background Risk 12-Jun-15 110 32%

Total Contingency 349 100%


Contribution of Each Risk to the Time Contingency (2)

26/

May

/16

20/

Jan/

16

01/

Dec

/15

28/

Oct

/15

1

6/O

ct/1

5

06/

Oct

/15

0

1/O

ct/1

5

01/

Oct

/15

3

0/Se

p/15

12/

Jun/

15

06/ Apr/ 15 15/ Jul/ 15 23/ Oct/ 15 31/ Jan/ 16 10/ May/ 16 18/ Aug/ 16 26/ Nov/ 16 06/ Mar/ 17

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Cum

ulat

ive

Pro

babi

lity

g g

Resources and Cost


Estimate is $6.86 billion

Cost Risk Results


$7,000,000,000 $8,000,000,000 $9,000,000,000


0

50

100

150

200

250

300

350

400

450

500

550

600

650

Hits

0% $6,189,493,454

5% $6,861,086,752

10% $7,007,130,725

15% $7,101,752,271

20% $7,188,049,098

25% $7,258,107,546

30% $7,331,963,666

35% $7,391,997,593

40% $7,452,396,460

45% $7,514,273,968

50% $7,568,332,545

55% $7,630,117,222

60% $7,687,432,085

65% $7,749,200,097

70% $7,813,239,105

75% $7,890,603,866

80% $7,976,966,619

85% $8,088,750,149

90% $8,234,937,806

95% $8,429,356,166

100% $9,283,008,684

Cum

ulat

ive

Freq

uenc

y

Spacecraft for PMChallenge 2009Entire Plan : Cost

Baseline $6.86 billion is only 5% likely

The 80th percentile (P-80) is $7.98 B for a $1.1 B contingency

Spread P-5 to P-95 is $6.86 to $8.43, for $1.57 B

Contribution of Each Risk to the Cost Contingency


Contributions of Individual Risks to Cost RiskAll Risks $ millionsCost Risk is based on immature data 287Funding from Congress is problematic 277Fabricaton requires new materials 168Schedule Estimate inaccurate 110Schedule for testing is aggressive 63New designs not yet proven 31Lost know‐how since last full spacecraft 2Several alternative designs considered 1Requirements have not been decided 0

Scatter Diagram of Time and Cost


3%

1% 94%

2%

20/Feb/15 31/May/15 08/Sep/15 17/Dec/15 26/Mar/16 04/Jul/16 12/Oct/16 20/Jan/17 30/Apr/17

Entire Plan: Finish

3%

$6,200,000,000

$6,400,000,000

$6,600,000,000

$6,800,000,000

$7,000,000,000

$7,200,000,000

$7,400,000,000

$7,600,000,000

$7,800,000,000

$8,000,000,000

$8,200,000,000

$8,400,000,000

$8,600,000,000

$8,800,000,000

$9,000,000,000

$9,200,000,000E

ntir

e Pl

an: C

ost

5%

Spacecraft for PMChallenge 2009

12/ Jun/ 15

$6,861,500,000

Deterministic Point Inside both limits Outside both limits

There is a 94% chance of overrunning both cost and scheduleNotice the slope of the scatter –time drives cost in this model

Probabilistic Cash Flow


Deterministic Cost: $6,861,500,000

Det

erm

inis

tic F

inis

h: 1

2/Ju

n/15

26/ Jul/ 08 11/ Feb/ 09 30/ Aug/ 09 18/ Mar/ 10 04/ Oct/ 10 22/ Apr/ 11 08/ Nov/ 11 26/ May/ 12 12/ Dec/ 12 30/ Jun/ 13 16/ Jan/ 14 04/ Aug/ 14 20/ Feb/ 15 08/ Sep/ 15 26/ Mar/ 16 12/ Oct/ 16 30/ Apr/ 17

Time

0

1,000,000,000

2,000,000,000

3,000,000,000

4,000,000,000

5,000,000,000

6,000,000,000

7,000,000,000

8,000,000,000

9,000,000,000

Cum

ulat

ive

Resource Flow for CostFilter: Entire Plan

Mean P20 P80

This monthly probabilistic cash flow can be compared to the budget to adjust the spending patterns when considering risk

Probabilistic Cash Flow Compared to Planned



Summary (1)

• The focus is on the risks, not their impact• Risks “explain” the need for a contingency• Management appreciates this focus on

risks• Risk interviews are conducted at 10,000

foot level, where people typically think of risk

• Interviews go faster, stick to the substance


Summary (2)

• Use Risk Register for quantitative analysis• Specific risks can be quantified and assigned to

schedule activities– Quantification is probability and impact– A risk can affect several activities– An activity can be affected by several risks

• Risk Factors can be combined with other more traditional approaches such as 3-point estimates for background risk or probabilistic branching

Summary (3)

• Schedule uncertainty creates cost uncertainty

• Analysis of cost simultaneously with time requires inserting the budget/resources into the schedule and simulating both together

• More accurate cost risk analysis and full appreciation of the role of schedule risk in creating cost risk


Using the Risk Register in Integrated Cost/Schedule Risk Analysis with Monte Carlo Simulation

David T. Hulett, Ph.D.Hulett & Associates, LLC

Dr. Dan Patterson, PMPAcumenpm

NASA PM ChallengeDaytona Beach, FL

February 24-25, 2009


patterson.dan

Economy & Finance

risk factors approach

risk mitigation

risk model

cost risk

risk ability

risk factorprobability

risk factors mechanics

risk factors module