9/09/2009 1

Risk Analysis

Bob SklarEngineering Fellow & Chief Engineer (Retired)Raytheon CompanyUniversity of Arizona SIE-554 Fall 2009

9/09/2009 2

Benefits of Risk Analysis

• Increases probability of program success– Potential to reduce program schedule & cost

• Facilitates proper allocation of resources– Helps in program planning

• Communicates program status / health– Informs program / company management– Informs customer

Effective risk analysis is not easy, but a process and tools are

available

Effective risk analysis is not easy, but a process and tools are

available

9/09/2009 3

Concept Design• System• Subsystems• Components

Preliminary DesignDetailed Design

Production Design

DESIGN Initial Analysis• Assessment• Management• Communication

Updated AnalysisUpdated Analysis

Updated Analysis

RISK ANALYSIS

Initial Plans• Components• Subsystems• System

Updated PlansUpdated Plans

Updated Plans

PROGRAM TEST PLANS

Risk Analysis is a Continuous Process

As the design evolves, risk analysis & test plans are

updated, leading to design improvements &

program success

As the design evolves, risk analysis & test plans are

updated, leading to design improvements &

program success

9/09/2009 4

Risk Topics

•What is Risk?

•Types of Risk

•Assessment of Risk

•Reporting & Tracking Risk

•Risk Management

•Risk Mitigation

•Example

9/09/2009 5

What is Risk? - 1

• From the dictionary:– The possibility of loss or injury.– The chance or possibility of incurring loss or

misfortune.– The uncertainty of future returns (financial risk).

• Risk is “the potential of an adverse condition occurring on a project which will cause the project to not meet expectations”

- University Information Services, Georgetown University

9/09/2009 6

What is Risk? - 2

“Risk is a measure of the potential harm or loss associated with an activity executed in an uncertain environment.”

-Terry Bahill

“Risk is Uncertainty”- Bob Sklar

Something that is unknown because it has not yet been designed is a design task; not a risk

Something that is unknown because it has not yet been designed is a design task; not a risk

9/09/2009 7

What is Risk Analysis? Risk analysis includes:

– Risk assessment involves identifying sources of potential harm, assessing the likelihood that harm will occur and the consequences if harm does occur.

– Risk management evaluates which risks identified in the risk assessment process require management and selects and implements the plans or actions that are required to ensure that those risks are controlled.

– Risk communication involves an interactive dialogue between stakeholders and risk assessors and risk managers which actively informs the other processes.

- Wikipedia

Risk analysis = risk assessment + risk management + risk communication

Risk analysis = risk assessment + risk management + risk communication

9/09/2009 8

• Performance Risk– Technical risks, normally tracked using technical

performance measures. Inability to achieve technical requirements.

• Schedule Risk– Unforeseen delays in completing tasks.

• Cost Risk– Unforeseen cost overruns, often associated with

performance and schedule risks.

• Project Risk– The potential of an adverse condition that will cause

the project to not meet customer expectations

Different Types of Risk

9/09/2009 9

Interrelationship of Risks

CostRisk

ScheduleRisk

PerformanceRisk

ProjectRisk

Schedule slips

Limited funds

Com

pressed schedule

Lim

ited

fund

s

SchedulesBudg

ets

Cos

t ove

rrun

s Schedule slips

Technical problem

s

Requirem

ents

Technical problemsTe

chnica

l pro

blem

sThese three

risks are often traded off.

The effect of a risk in one category may become the source of risk in another category. The most serious of the low-level performance, schedule and cost risks will be elevated to become project risks.

Major performance, cost & schedule risks work their way up to the overall project

Major performance, cost & schedule risks work their way up to the overall project

9/09/2009 10

• Low Risk– Technology mature and can be applied

operationally. Multiple technology options available.

• Moderate Risk– Technology development required before

operational application. Few technology options exist.

• High Risk– Technology immature. Extensive development

needed even before laboratory demonstration. Only one plausible technology option exists.

Evaluating Performance Risk

9/09/2009 11

• Suppose we conduct a test and discover that a subsystem does not meet a key requirement

• Since the uncertainty has been eliminated, has risk also been eliminated?

• No, there may be a high level of project risk caused by not meeting this requirement

• Therefore, the subsystem design must be fixed (unless the overall system can somehow accommodate this performance problem)

Is Elimination of Uncertainty Sufficient?

It is not enough to eliminate uncertainty; program requirements must still be satisfied It is not enough to eliminate uncertainty;

program requirements must still be satisfied

9/09/2009 12

• Accepting risk is desirable if:– It offers the potential of large payoffs in

performance, cost or schedule• Normally, risk buys performance potential• Sometimes, reduced cost or schedule is the objective• Rarely do all 3 benefit from accepting risk

– The risk can be managed• Risk management plan is critical & must be seriously

implemented

– The risk is accepted by the customer• Must understand risk and potential consequences• Risks don’t always pay off• Example: Use of unproven technology could provide

performance improvement if successful; more cost & delayed schedule if not

Accepting Risk Can Be Good

9/09/2009 13

“Fear of some harm ought to be proportional not only to the magnitude of the harm, but also to the probability of the event.”

- Logic, or the Art of Thinking

Antoine Arnauld, 1662

We refer to these as:

1. Severity of Consequence

2. Relative Likelihood of Occurrence

Risk has 2 Components

For convenience, the product of these 2 risk components is often used as an overall measure of risk

For convenience, the product of these 2 risk components is often used as an overall measure of risk

9/09/2009 14

• Since risk represents uncertainty, its components are difficult to explicitly quantify

• Engineers with appropriate expertise can use their experience and judgment to estimate relative risk levels

• Relative risk levels, while not precise probabilities, provide a means of tracking and communicating areas needing special attention

How do We Quantify Risk?

Risk assessments highlight areas demanding special attention

Risk assessments highlight areas demanding special attention

9/09/2009 15

Legend1 = 2 =3 =4 =5 =6 =7 =

(Brie

f des

crip

tions

)

Reporting & Tracking Risk

Report Date:

Arrows indicate changes from

previous report

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

Potential Severity of Consequence

Rel

ativ

e L

ikel

iho

od

of

Occ

urr

ence HighModerateLow

6

7

5

1

4

2

3

The goal is to reduce all risks to low levels during the development process

The goal is to reduce all risks to low levels during the development process

9/09/2009 16

Re

lati

ve

Lik

eli

ho

od

of

Oc

cu

rre

nc

e

Potential Severity of Consequence0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

1.0

0.90.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

HighModerateLow6

7

51

4

23

Some Observations on Risk Plot

•The ordinate (vertical axis) is not probability•Probabilities cannot be calculated because of uncertainty and unknown unknowns

•This axis is labeled relative likelihood– Relative because it concerns relationships

between risks – Likelihood not in the

mathematical sense,

but in the dictionary

sense indicating risks

that are most likely

9/09/2009 17

•Estimated risks could increase or decrease due to

– Design changes– Analysis or test results– Other new information

What Causes Risks to Change?

The design evolves; knowledge evolves – always look for new risks & better

estimates for previously known risks

The design evolves; knowledge evolves – always look for new risks & better

estimates for previously known risks

9/09/2009 18

1 N

eglig

ible

2

Unl

ikel

y3

Li

kely

4 H

ighl

y P

rob

able

5 N

ear

Cer

tain

ty

Lik

elih

oo

d

1 Marginal

2 Significant

3 Serious

4 Very Serious

5 Catastrophic

Consequence

High Risk

Moderate Risk

Low Risk

Alternate Form of Risk Plot

Some customers prefer this form

Some customers prefer this form

9/09/2009 19

1 N

eglig

ible

2

Unl

ikel

y3

Li

kely

4 H

ighl

y P

rob

able

5 N

ear

Cer

tain

ty

Lik

elih

oo

d

1 Marginal

2 Significant

3 Serious

4 Very Serious

5 Catastrophic

Consequence

High Risk

Moderate Risk

Low Risk

Alternate Form of Risk Plot

Some customers prefer this form - but it has the disadvantage that risks with

potential catastrophic

consequences can never be

retired

Some customers prefer this form - but it has the disadvantage that risks with

potential catastrophic

consequences can never be

retired

9/09/2009 20

Preferred Form of Risk Plot

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0Rel

ativ

e L

ikel

iho

od

of

Occ

urr

ence

Potential Severity of Consequence

HighModerateLow

This is basically a communications aid, so do it the best way for your customer

This is basically a communications aid, so do it the best way for your customer

9/09/2009 21

“Good risk management will not prevent bad things from happening. But when bad things happen, good risk management will have anticipated them and will reduce their negative effects.”

-Terry Bahill

Performance, schedule and cost risks at the lower (subsystem) levels can often be accommodated at the project level by reallocating resources or requirements, but this requires that program management be aware of problems early.

Risk Management

9/09/2009 22

• Develop comprehensive risk management plan– Risks identified; mitigation plans developed

• By engineers with related expertise

• Assign each risk to a responsible engineer– In design area impacted by this risk

– Manages risk mitigation plan; reports status

• Conduct tests at all levels as early as possible– Uncover design problems in time to fix deficiencies

– Validate or refine analytic models (confidence)

• Keep customer informed– Focus on moderate and high risks; report status &

program impact

Ways to Manage Risk

9/09/2009 23

• Develop program plan, schedule, and budgets that can accommodate risk

– Test intensive (all levels & early; prototypes)

– Spare parts (to preserve schedule if failures)

– Back-up plans (work-arounds)

– Anticipate failures

– Ample design margins

• Continually search for new risks as knowledge evolves over time

Risk Mitigation

The program plan must tradeoff performance risk and the cost of additional tests

The program plan must tradeoff performance risk and the cost of additional tests

9/09/2009 24

• For each high or moderate risk area, the plan should include:

– Analysis, simulations, tests and other activities capable of progressively reducing uncertainty

• Plan for each activity should include detailed schedule and required resources (including personnel)

– Activities should be conducted as soon as allowed by available hardware/software

– Reassess risks based on results of activities

– Have backup plan in case activity fails

Risk Mitigation Plan Development

Risk management is an integral part of the overall program plan. It is a continuous & iterative process.Risk management is an integral part of the overall program plan. It is a continuous & iterative process.

9/09/2009 25

Hig

hM

od

erat

eL

ow

Ris

k

CY07 CY08 CY09

Analysis & Preliminary Simulation

Updated Dynamic

Simulation

Initial Integration

(Prototype Performance Test)

Final Integration

(Final Performance Test)

Mitigation Plan (Risk Waterfall Chart)Risk 352: Mechanical integrationSubsystem: Staging AssemblyResponsible Engineer: John Stager

Every risk item is assigned to a specific responsible design engineer

Every risk item is assigned to a specific responsible design engineer

Plan should include detailed descriptions of each mitigation activity

Initial Mechanical Fit Check

Activities are introduced to progressively reduce risk

Activities are introduced to progressively reduce risk

9/09/2009 26

• If you are the responsible engineer for a missile subsystem and need to make a risk status presentation at the monthly meeting with your customer, consider presenting the following:

– Overview – list key changes from last month

– Updated risk register• Subsystem risks & risk assessments; highlight changes

– Risk plot (likelihood vs. severity)• Indicate changes from previous month

– Risk waterfall charts for high/medium risks• Indicate changes from previous month

– Results of any risk mitigation activities completed in last month, justifying risk assessment changes

Reporting Risk

9/09/2009 27

Risk Analysis Example

For this risk example, we will address the MK104 Dual-Thrust Rocket Motor (2nd stage)

Public Released Raytheon Images

Example intended to illustrate risk analysis process, not actual SM-3 history

9/09/2009 28

• Initial design trades and at least preliminary subsystem designs have been completed

–Rocket motor size (consistent with other missile stages)

–Case materials–Propellant type (solid)–Propellant design (thrust profile)–Stage interfaces (mechanical/electrical)

Some Design Status Assumptions

Design Challenges are tasks, not risksDesign Challenges are tasks, not risks

9/09/2009 29

Risk No. Risk Consequences Likelihood Severity Risk Factor

1 May not achieve specified total impulse

Overall missile performance may not be acceptable

0.80 0.70 0.56

2 2nd thrust may not initiate properly

Would reduce mission effectiveness 0.40 0.80 0.32

3 May exceed weight allowance Overall missile performance may not be acceptable

0.50 0.60 0.30

4 Mechanical interfaces with other stages may be inadequate

Missile failure during transportation, launch or flight

0.20 0.90 0.18

5 Stage production cost may exceed allowance

Total missile cost may not be acceptable

0.50 0.30 0.15

6 Propulsion development time may exceed plan

Development time (and associated cost) may not meet program objectives

0.70 0.30 0.21

7 Propulsion stage may not meet safety requirements

Missile may not be acceptable to customer

0.30 0.80 0.24

Risk Register Example

Risk Factor = Likelihood x Severity

2nd Stage Rocket Motor

9/09/2009 30

Legend1 = Total impulse2 = 2nd thrust3 = Stage weight4 = Interfaces5 = Stage cost6 = Schedule7 = Safety

Example Risk Assessment

Report Date: 1/15/2007

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0Rel

ativ

e L

ikel

iho

od

of

Occ

urr

ence

Potential Severity of Consequence

HighModerateLow

6

7

5

1

4

2

3

These estimates, based on judgment, may not reflect realityThese estimates, based on judgment, may not reflect reality

2nd Stage Rocket Motor

9/09/2009 31

Example Risk Waterfall Chart

Each high or moderate risk needs a mitigation planEach high or moderate risk needs a mitigation plan

Hig

hM

od

erat

eL

ow

Ris

k

CY08 CY09 CY10

Heavyweight Motor Combustion Test

(Prototype Performance Test) Environmental Tests

Subsystem: 2nd Stage Rocket MotorRisk 1 (may not achieve specified total impulse)Responsible Engineer: Mary Burns

Descriptions of each mitigation activity should be included as part of the plan (with detailed description, schedule, required resources, etc.)

Propellant & Igniter Tests

Lightweight Motor Combustion Test

(Final Performance Test)

9/09/2009 32

Some Final Thoughts

“The parts of risk analysis we observe are only the communication tools. That is why they don’t have to be precise. Real risk analysis takes place at lower levels, where design & test engineers are busy at their work. Systems engineers facilitate this process.”

“Risks cannot really be controlled – bad things happen. But the anticipation of risks and the control of their consequences can indeed be managed.”

- Bob Sklar