copyright © 1995-2004, dennis j. frailey, all rights reserved day 3, part 3, page 1 1/11/2004 day...

Copyright © 1995-2004, Dennis J. Frailey, All Rights Reserved Day 3, Part 3, 1/11/2004

Day 3, Part 3

Software Risk Management


Outline

• Overview of the Risk Management Process

• Risk Assessment

• Risk Control

• Risk Examples

• The Risk Management Plan


The Overall Planning Cycle

AnalyzeJob

Manage Risks

Execute

GenerateDetailed Plans

GenerateInitial Plans

Measure, Manage Productivity and Quality


Software Management:A Framework for Risk

Management

SoftwareDevelopment

RiskManagement

ProjectManagement


Another Model of theRisk Management Process

RiskManagement

Plan

Execute

Measure

EngineerQuality


What is a Risk?• Something that might go wrong with

the project, the product or the process• When stating a risk, be sure indicate

the problem and the cause– “The project might be late”

This doesn’t say much. Why might it be late?

– “There might be employee turnover”So what? This states the cause but not the

problem

– “The project might be late due to employee turnover”Good. This states both the cause and the

problem


Other Examples

• “Lack of familiarity with the language” < states a cause but no problem

• “Incorrect software due to lack of familiarity with the language” < states a product problem and a cause

• “Fail to meet schedule due to lack of familiarity with the language” < states a project problem and a cause


The Risk Management Process:

1) Risk Assessment (4 activities)2) Risk Control (2 activities)


1) Risk Assessment(This is Done as part of planning)

A) Risk Identification- What are the risks?

B) Risk Analysis- What is the likelihood & impact?

C) Risk Prioritization- Which risks are most serious?

D) Risk Planning & Mitigation- Minimizing impact

Copyright © 1995-2004, Dennis J. Frailey, All Rights ReservedDay 3, Part 3, 1/11/2004

Risk Management Plan Should Indicate …

• What process you have already followed to identify, analyze, prioritize, and mitigate risks– What risks you have identified

• And the evidence that you base this on

– How you have analyzed these risks– How you have prioritized them– How you have mitigated them


2) Risk Control(This is done as part of project

execution)

A) Risk Monitoring

- Watching to see if risks happen

B) Risk Abatement

- Counteracting risks

- Taking contingency actions


Risk Management Plan Should Indicate …

• What process you will follow during the project to control risks– How you will monitor them (this usually

ties strongly to your measurement plan)

– How you will abate risks (contingency plans, ongoing mitigation)

• And what process you will use to keep the plan up to date– Ongoing assessment, updating of

plans, priorities, etc.


Risk Management Methods

The following methods will be discussed:

1) Barry Boehm’s Method– Widely known– Very pragmatic approach– Combines assessment with control

2) Dennis Frailey’s Method– Somewhat more comprehensive– Derived from DoD standards for

defense system software development


Top 10 Risks on

Project Alpha

1. Staffing

2. Late Hardware

3. Requirements Definition

4. Real-time perf-

Boehm’s Method ofRisk Management (5 steps)

Risk Assessment (steps 1-2):

1) Identify the top 10 risk items (identification, analysis and prioritization)

2) Present a plan to resolve each of the top 10 items (mitigation; planning for control)


Boehm’s Method (continued)

Risk Control (steps 3-5):

3) Update the list and the plan monthly (monitoring)

4) Highlight the risk items at monthly project reviews (monitoring)

5) Initiate corrective action for risks that occur (abatement)

[6) Follow-up until the issue is resolved]


Frailey’s Method of Risk Management (11 steps)

Risk Assessment (steps 1-7)– Identification (Step 1)–Analysis (Step 2- 4)–Prioritization (Step 5)–Planning (Step 6,7)

Risk Control (steps 8-11)–Monitor (Step 8, 10)–Abatement (Step 9)–Planning (Step 11)


Risk Assessment

Frailey Method (steps 1-7)


1) Identify all Risk Elements(risk identification)

• Risk Elements consist of:– things that can go wrong – patterns of risk change over the

lifecycle• for example, cost estimating risks occur

early, whereas risks of staff burnout occur later

• If it has already happened, or is certain to happen, it is a problem, not a risk!– You should be discussing your action

plan for managing the problem


Notes

• Most risk identification is performed as a part of other planning processes (see the previous chapters in these notes)

• But it is also good to have a pro-active attempt to identify all risks in case something “fell through the cracks”


Risk Identification Meeting

• A supplementary meeting to identify risks– Some may have been overlooked– The meeting also helps to focus

attention on risk issues

• Identify the patterns of risk– Risk patterns change over the lifecycle– For example, cost estimating risks occur

early, whereas risks of staff burnout occur later


Recall How All Planning Activities Identify Risks

AnalyzeJob

Manage Risks

Execute





2) Partition into Categories(Analysis)

• Sample Categories:– Cost risks– Schedule risks– Other management risks– Technical risks– Other risks specific to the situation, such as

safety or security risks

• One Risk may have multiple categories– Estimating inaccuracies can lead to cost and

schedule risks


Why Partition Into Categories?

• Risks may need to be prioritized as demanded by the situation– “continue at any cost” vs. “only do if low

cost”

• Different categories of risks may require different mitigation approaches– Technical risks may require performance

analysis– Schedule risks may require process

changes


Other Reasons to Partition Risks into Categories• Different people may be concerned

about different risks– Technical lead vs. – Finance manager vs. – End user waiting for delivery

• Different people may be responsible for different risks


3) Identify Contributing Factors

(Analysis)• Many risks can occur in several

ways

• If you aren’t careful, you will only be looking for one of the ways


Example of Multiple Contributing Factors

Risk: Not enough memory to hold the softwarePossible Contributing Factors (Causes): Size of computer memory Expertise of programming staff Efficiency of compiler Choice of algorithms Operating system


Using a Hierarchy of Contributing

Factors• Each risk can be seen as a

contributing factor to a larger risk• The top level risk is that the project

will fail• Sometimes it helps to use a

hierarchy to organize risks and contributing factors

• (See next slide)


A Sample Risk Hierarchy

Staffing Funding . . .

ProcessorToo Slow

. . .

Size ofMemory

ProgrammingExperience

CompilerEfficiency

Choice ofAlgorithms

MemoryToo Small

PerformanceFailure

ProjectFailure


4) Identify Potential Risk Monitoring & Mitigation

Plans (Analysis)

• This must be done for each contributing factor

• See next slides for examples


Potential Risk Mitigation Plan

Risk: memory size inadequateFactor: Compiler produces bloated codePotential mitigation:

•Choose a more efficient compiler•Negotiate improvements with vendor

Factor: Inexperienced programmersPotential mitigation:

•Training program •Use more experienced programming staff


Multiple Risks with One Approach

Risk: memory size inadequateFactor: Compiler produces bloated code

Factor: Inexperienced programmers

Potential mitigation that applies to both:•Select a larger memory size

Potential monitoring that applies to both:•Track size estimates monthly•Update at each major milestone


5) Rank and Prioritize Each Risk

(Prioritization)• Prioritize on the basis of probability

(how likely) and impact

You cannot prevent all risks - focus on the big onesYou cannot prevent all risks - focus on the big ones

Risk Likelihood Cost Weighted Cost

Late Hardware 75% 100,000 75,000

Sub-Contractor Failure 20% 250,000 50,000

Memory Size 50% 50,000 25,000

Test Equipment Delay 30% 40,000 12,000

Requirements Changes 99% 5,000 4,950

Building hit by plane 0% 50,000,000 50


6) Identify Monitoring Procedures for Each Risk

(Planning for control)•Determine how to tell if it is a problem;

how frequently to monitor; etc.•Example: monitor projected size vs.

memory limits on a monthly basis

0

50

100

150

J an Feb Mar Apr May J un J ul Aug Sep Oct Nov Dec

Limit Threshold Estimate


7) Develop a Contingency Plan(Planning)

• Identify what to do if the risk occurs despite your mitigation efforts

Risk: memory size exceededContingency Plan:

• Switch to a slower but smaller algorithm• Use a more efficient compiler • Use a smaller operating system• Use larger memory size


Risk Control

Frailey Method (steps 8-11)


Review Status and Take Action (Steps 8, 9)

8) Review status of risks at periodic reviews (Monitor)– Metrics– Changes in impact analysis

9) Take appropriate action when called for (Abatement)– Closer monitoring– Contingency activities


“Do Your Homework”(Steps 10, 11)

10) Track all actions to closure (Monitoring)– Don’t forget about them

11) Update the plan (Planning)– Keep it consistent with current knowledge and

status


Whatever Method You Use

• It should be possible to see a thread for each identified risk:– Risk– Risk Factors / Causes (there may be

many)– Evidence– Priority– Mitigation– Monitoring– Contingency


Beware of Subcontractors and Co-contractors

•Risk management applies to these as well

•Include risk management elements in contracts We want to

monitor your risks

Just trust us


Risk Assessment


Risk Assessment• Goal: understand what the risks are,

what they mean, and how best to manage them

• Method: develop a risk management plan– Documents your risks and your plans for

managing them– Communicates responsibilities to all

affected parties• Update with changes in current risks &

their impact/likelihood of occurrence


Risk IdentificationThe First Step of Risk Assessment

• We have seen how all management activities, especially planning activities, serve to identify risks

• It helps if you know what to look for• Several authors have talked about

risks associated with software development


Boehm’s Top Ten Software Risks

• This is Boehm’s list, based on experience with many projects in the 1960-1980 time period

• He also provides management techniques for mitigating them ...

• Your project must define its own list• But this is a good place to start

looking


1) Personnel Shortfalls

• Use top talent• Use people who are well matched to

the problem (i.e., they know the application, tools, etc.)

• Pre-schedule the key people• Cross training

– So you don’t depend on heroes

• Team building


• Use good estimating techniques– Know before you commit

• Incremental development cycles

• Detailed milestones within each phase– Spot trouble early– Provide evidence of progress

• Software reuse– Don’t build what was built before

2) Unrealistic Schedules & Budgets


3) Developing the Wrong Software Functions

• Mission analysis– Understand what is supposed to happen

• User surveys and communication– Know what the user needs expects

• Prototyping– Try it out

• Document the end user needs early in the project & use as requirements documents


4) Developing the Wrong User Interface

• Prototyping– Involve the user in trying it out

• Task analysis

• Scenario models

• etc,


5) Gold Plating• Requirements scrubbing

– Don’t build what is not required• Cost-benefit analysis

– Determine what counts the most• Design-to-cost

– Development costs include debugging, error correction

• Design to life-cycle-cost– Life cycle costs include maintenance,

debugging, etc.


6) Continuing Stream of Requirements Changes

• Establish a high change threshold – Make it costly to make a change

• Information hiding• Incremental development

– Do the stable requirements first• Establish bounds for requirements• Monitor requirements stability &

completion– Know the risk of proceeding


7) Shortfalls in Purchased Software/Hardware

• Benchmarking– Learn what is the best

• Inspections– Make sure it is in good shape before you buy– Reference Checking– Are they able to do the work?– Do they deliver?

• Compatibility Analysis– Make sure it fits with your standards, tools,

documentation requirements, etc.


8) Shortfalls in Externally-Performed Tasks

• Subcontract management• Reference checking• Pre-award audits and capability

evaluations• Award fee contracts

– Reward or bonus if satisfied• Competitive design or prototyping• Team building

– Make them want you to succeed


9) Real-time Performance Shortfalls

• Simulation• Benchmarking• Modeling• Prototyping• Instrumentation• Tuning


10) Straining the Limits of Computer Science

• Technical analysis

• Cost-benefit analysis

• Prototyping

• Reference Checking


Risk Analysis & Prioritization• Which ones are Important?• A popular method is to compute two

numbers for each risk:– How likely is it to happen (a probability

from 0 to 1)– How much will it cost if it happens

(dollars)• Then make a table showing risks,

likelihood, cost, and weighted cost (likelihood * cost)


Risk Analysis TableRisk Likelihood Cost Weighted Cost



Late Hardware 75% 100,000 75,000



Memory Size 50% 50,000 25,000


Prioritized Risk Analysis Table


Late Hardware 75% 100,000 75,000


Memory Size 50% 50,000 25,000





Alternative Risk Analysis Table

Risk Likelihood I mpact Weighted

Late Hardware 7 6 42

Sub-Contractor Failure 2 7 14

Memory Size 5 4 20

Test Equipment Delay 3 3 9

Requirements Changes 8 2 16

Building hit by plane 1 9 9

Note that this method produced a different ordering, but the extreme cases came out in the same order.


WARNINGThis method of risk prioritization has

many risks itself!• Not all risks can be quantified in terms

of dollar impact• Estimates of impact are very subjective• Impacts change over time -- must be

revisited• Risk mitigation techniques may have

risks of their own


Risk Mitigation


Risk Mitigation

• Taking actions to reduce the likelihood or net impact of a risk

• We saw many such actions in reviewing Boehm’s list of risks

• Each potential action must be evaluated in terms of its cost vs. the potential impact


This is the Step where you Impact the Planning Process

AnalyzeJob

Manage Risks

Execute





Mitigation Example

Risk: Subcontractor will fail to deliver

Weighted Cost: $50,000

Mitigation Options:•Do it in-house -- Costs $100,000 extra•Send staff to live with subcontractor -- $50,000•Monthly visits -- $12,000


Risk Table After MitigationRisk Likelihood Cost Weighted Cost

Late Hardware 75% 100,000 75,000

Memory Size 50% 50,000 25,000

Sub-Contractor Failure 5% 250,000 12,500+12,000



Building hit by plane 0.0001% 50,000,000 50


Risk Plan May Include Tables of Mitigation, Contingency,

etc.Methods

Risks

Memory Size

Benchmark

ExtraHardware

Backup

etc.Prototype

SubcontractorDelayed

Late Test Equip.

etc.

Late Hardware X

X

X

X

X

X


Risk Control


Risk Control

• Monitoring– Watch what is happening

– Watch for signs of danger

• Risk Abatement– Applying contingency plans

– Minimizing impact


Methods of Risk Monitoring• Reviews

– Periodic status reports– Must be honest reviews, not “dog and

pony shows”

• Metrics – Data to compare actuals with plans and

past performance


What to Monitor

• All high priority risk items• Consider cost of monitoring

– Monitor only what is worthwhile

• Consider the changes caused by measurement to the organization & the process


How Often to Monitor

• It depends on priority - you must plan. For example:– Critical items daily or weekly– Normal items weekly or monthly– Minor items quarterly

• Monitoring too often costs money and slows down the process


Watch Known Danger Points

• Monitor status at key milestones or progress points– Are we where we should be by

now?

We always have a flurry of changes prior to CDR


Don’t Monitor Too Often

• Demotivates people

• Costs a lot

• Robs resources from productive activities


Don’t Monitor in Too Much Detail (overly precise)

• Costs a lot• Does not give significant additional

information• May generate a lot of misleading

numbers– Late by 2.7321 weeks is not much

different from late by 3 weeks


Things to Watch Out For - IHiding the facts to save face

– The purpose of monitoring is to manage properly, not to find fault with individuals

– Individuals must trust that you will use the metrics properly to fix the process, not to punish the messengers

– One solution is self-measurement -- have the individuals measure and monitor themselves without communicating higher except on an aggregate scale or with big problems that they cannot handle


Things to Watch Out For - II

Hiding the facts to save the project–Egos and jobs of technical staff vs.

judgment of management vs. pocketbook of sponsor

–This can get very political and very complex


Things to Watch Out For - III

Failure to get the data because of fear, overconfidence, or other psychological factors–Be careful of human nature–Focus on teaming, responsibility, and

professional behavior


Things to Watch Out For - IV

Failing to get the data because of high overhead– Automate collection– Consider using a separate metrics

collection/analysis staff to minimize impact on development staff • but not if it destroys teamwork


Things to Watch Out For - V

Misinterpretation of data–Any number can be interpreted in many

ways• The “three ways to get it wrong” rule

–Be prepared to ask lots of questions–Define standard measures and methods

of graphing data to minimize unintentional misinterpretation

–Educate everyone in statistics


Things to Watch Out For - VI

Failure to trust past performance–Your “track record” is your most

reliable indicator


The Learning Process

Information is Communicated

Information isReceived

Information isAnalyzedAction

Each step offers many opportunities for errorEach step offers many opportunities for error


Example of Misinterpretation

Productivity isLow

Not EnoughWork Being

Done

Not WorkingHard Enough

MoreOvertime

Perhaps the Real Problem is Too Much Overtime Already

Perhaps the Real Problem is Too Much Overtime Already


Rate Chart - Actual vs. Plan• Shows actual vs. planned progress

for some artifact being produced– Coded units– Tested units– Inspected units– Specifications – Requirements defined– Requirements tested– etc.

• Rate charts work for anything that has discrete, measurable units

Units Tested

0

20

40

60

80

100

1 2 3 4 5 6 7 8 9 10 11

Plan

Actual

History


Risk Abatement• Establish thresholds so you know when

to act– Beware of the “frog in the water” problem

– Historical experience is a good basis to judge when things are getting out of hand


Risk Abatement (continued)

• Act promptly when necessary– Establish action plans before they are

needed – Learn the plan (fire drills) -- there may

be no time in an emergency

• Use backup plans if needed– Develop them during the planning

phase– Practice them too!


Risk Abatement (continued)

•Let the team participate in the planning–Their cooperation is necessary for

success–Their ownership of the plan will

improve chances of cooperation


The Risk Management Plan


Risk Management Plan

• A documented plan that communicates the risks and the plans for managing them to everyone concerned

• It also helps everyone know what to do during a crisis

• The plan should be reviewed and updated from time to time


Goals of a Risk Management Plan

• To show all concerned that you …… understand the risks of the project… know how to manage those risks… have taken appropriate actions to

mitigate risks… have plans in place to monitor those

risks

The purpose of the risk management plan is not to deny that you have risks, to avoid mentioning

important risks, or to belittle the importance of key project risks.

The purpose of the risk management plan is not to deny that you have risks, to avoid mentioning

important risks, or to belittle the importance of key project risks.


Contents of aRisk Management Plan

• The risk management process to be used during execution of the project

• Results of initial risk analysis– Analysis and priorities– Key risks identified– Actions taken to mitigate key risks

• Minimize impact• Reduce likelihood

• Monitoring and abatement plans


Sample Process Description• We will hold a monthly meeting

attended by … at which we will– Review actions from previous meetings– Review the status of the top 10 risks– Re-prioritize the risks– Assign actions as required

• Roles during this meeting– Record keeper– Facilitator– Etc.

• We will revisit the plan and consider an update every … months


Sample Chart ShowingRisk Analysis and Prioritization

Be sure to explain WHY each risk is important – provide EVIDENCE that it is as likely or as

expensive as indicated.

Be sure to explain WHY each risk is important – provide EVIDENCE that it is as likely or as

expensive as indicated.


Late Hardware 75% 100,000 75,000


Memory Size 50% 50,000 25,000





Sample Risk Occurrence Chart

Etc.Etc.Etc.Etc.Etc.Etc.Etc.Etc.

Etc.MedMedMedHiHiHiSpace too small

Etc.MedMedMedLowLowLowMemory too small

Etc.HiHiMedMedLowLowTurnover

…JMAMFJ

LikelihoodRisk

A chart like this helps you know when to expect certain risks to be more likely.

A chart like this helps you know when to expect certain risks to be more likely.


Sample Description of a Risk

Risk: high turnover of key staffEvidence: • recent turnover rates are 22%, • competition from growing

companies in townPriority: Red (urgent)• Likelihood is 75%• Impact is high


Sample Description of a Risk (continued)

Mitigation Plans and Actions: • Increased salaries 15%• Giving employees a bonus for successful

completion on timeMonitoring:• Will monitor project turnover rate monthly

and take action if exceeds 10%Contingency plans:• Increased hiring rates, use of contract

labor


Chart Showing Links to Metrics

Etc.Etc.Etc.Etc.Rqmts

Etc.Etc.Etc.Etc.Cost

Etc.Etc.Etc.Etc.Schedule

Etc.Etc.Etc.Etc.Memory

Etc.Etc.Etc.Etc.Staffing

Etc.Etc.Etc.Etc.Morale

Earned Value

Turnover

Size

MetricsRisks


Opportunities• An opportunity is the opposite of a

risk– Example: we might relieve schedule

pressure by reusing the data communication modules from another project

• Opportunities can be analyzed the same way that risks can

• Many organizations to risk / opportunity planning, analysis, etc. together as one activity


SummaryRisk Management

1) Risk Assessment – Done as part of project planning– Continues throughout the project– Includes planning for risk control

2) Risk Control – Done as part of project execution– You must respond promptly when

monitoring indicates a problem

A risk management plan is an important part of planning

A risk management plan is an important part of planning


ReferencesRisk Management

• Boehm, Barry, Software Risk Management, IEEE Computer Society Press, 1989, ISBN 0-8186-8906-4

• Jones, Capers, Assessment and Control of Software Risks, Yourdon Press, 1994.