+
+
Quantitative Risk Analysis for E& C Companies - the Long and Winding Road
June 2018
ECRI Sponsors Meeting
Quantitative Risk Analysis – the long and winding road
CONTENTS
• QRA in the E&C Industry
• QRA Models
• Parametric and Hybrid Models
• The Future
QRA IN THE E&C INDUSTRYQRA use in the E&C industry is quite widespread and typically used to support major or significant decisions. Some examples include:
• Technical Risk QRA’s
• Cause and Consequence analysis, FN curves, fault tree, event tree, bowties
• Reliability analysis – process plants
• Fire risk analysis
• Functional Safety
• Natural hazards risk analysis
• Schedule and cost risk analysis – our focus today
• Procurement, Contract and supply chain risk analysis
• Operating cost and ramp-up risk analysis
• NPV Risk Analysis, Real Options Analysis (ROA)
• Business Plan, Portfolio risk analysis and Bid risk analysis
• Decision Tree analysis
• Insurance - probable max loss analysis
QRA MODELS – WHAT MUST THEY ANSWER?Similar to any effective risk management exercise, the QRA must answer the cardinal risk management questions and support decisions.
Driving Actions and
Decisions• What are the major threats,
opportunities and uncertainties?
• How big are they?
• What are they ‘sensitive to’ &
how can we modify them?
• Are the risk profiles acceptable?
• Therefore what should
we do – and by when?
• What do we need to monitor &
review regularly in the execution
phase?
COMMUNICATION IS CRITCAL
QRA MODELS – WHAT CAN HAPPEN IF WE DON’T GET IT RIGHT The impacts of not understanding uncertainty and getting the numbers right can be catastrophic – there are plenty of examples.
Airport Link Tunnel Project:
• Authority approvals and access – resulted in delays and additional costs
• Difficult and variable conditions impacted tunnelling performance and productivity
• Unprecedented wet weather – October to January resulting in delays and extra costs
• Complexity of design resulted in more than doubling the tendered design allowance
• Construction complexity has resulted in extra materials e.g.
• Reinforcing steel increased from 86,000t to 133,000t
• Concrete increased from 820,000 m3 to 945,000m3
QRA MODELS – THE BENEFITS ARE TANGIBLEIndependent Project Analysis (IPA) research shows that actual schedules are shorter when risk analysis is used and that projects that use no risk analysis experience greater and more variable cost growth.
KEY TAKE AWAY – the effort to undertake QRA’s can
deliver benefits and be a CSF for your project
QRA MODELS – BAD PRACTICEWith software advances it is easy for any “fool to use them” and get bad decisions faster. Examples of some common practices that are wrong to be aware of include:
• Risk “bolted” on to estimates line by line without understanding uncertainty
• Risk “bolted” on to schedules without being able to build simple schedules well
• Bolted on to risk registers knowing the outcome will be wrong and not understanding
relationships between risks
• Sometimes a qualitative or deterministic analysis will suffice and be appropriate
• Diving into risk models without exploring and understanding uncertainty – modelling effort
should only be 10% to 20% of the total effort
• Not actually using a risk management process – the process is often more important than the
model (we sometimes hear the phrase “we need to do a quick Monte Carlo to determine our
risk”)
• Use of software without appropriate risk modelling skills and experience
QRA MODELS - BY PROJECT PHASEQRA’S can be and are often done across project phases – need to be clear about the context including information requirements and decisions at each gate that rely on the risk analysis outputs.
FEL1 STUDY
Risk:
- Fatal flaws identified
- Key risks identified
- Are controls practical
- Are risk treatments
practical
Estimate - High level, factored
- benchmarked
Schedule: - High level only
FEL 2 STUDY
Risk:
- Risk register & profiles for options
- Technical risk – hazard studies
- QRA for options
- Risk based decision to select best option
- Risk Treatment Plans for selected option
- QRA to assess contingency / accuracy
practical
Estimate - Option estimates
- Detail, factored and database
- Contingency/accuracy from QRA
Schedule: - Semi detailed for options
FEL 3 STUDY
Risk:
- Detail project risk register & profile
- Technical risk - HAZOP’s etc
- Detailed Risk Treatment Plans
- QRA to assess contingency / accuracy
- Risk Plan for Execution
Estimate - Detail Estimate
- Contingency/accuracy from QRA
Schedule: - Detailed schedule
- QRA to assess profile and time
contingency
FEL 4 EXECUTION
Risk:
- Project Risk Mgt Plan
- Technical risk – final HAZOP’s
- Detailed Risk Treatment Plans
- QRA reviews to assess profiles
- Execution risk assessments/change mgt
Estimate - Reviews and updates
- change mgt
- Reporting
Schedule: - Reviews and updates
- change mgt
- Reporting
Plans developed for next phase for review at the Gate, communication, consultation, monitoring and review throughout
KEY INTEGRATED RISK, COST and SCHEDULE ACTIVITIES
FEL - 1 FEL - 2 FEL - 3
Front End Loading Phases
ProjectExecution
PlanningStudy
FeasibilityStudy
Conceptual/Pre-Feasibility
Study
Gate
1P
roje
ct O
ptio
ns
Gate
2V
iab
le (Y
es/N
o)
Gate
3P
roje
ct E
xec P
lan
FEL-4
Pro
ject
Imple
menta
tion
KEY TAKE AWAY – each project phase has specific risk requirements and there needs to be
clear objectives and decisions for using quantitative risk analysis
QRA MODELS - RISK MODEL FRAMEWORKGood models help to understand how the project works, the effect of uncertainty and the implications of decisions.
KEY TAKE AWAY – need to spend time exploring and understanding project uncertainty and its
sources and how it may impact project decisions before getting immersed in the model.
QRA MODELS – REQUIRES SIGNIFICANT EFFORTQRA is done because the numbers are important and there is a major decision. It takes significant effort to plan and execute the whole process.
As an example, the process completed for a recent megaproject consisted of the following activities:
• Initial planning and project familiarization with the project team and interviews to understand the key uncertainties. Initial discussions on risk model structures. (1 week)
• Detail planning including (3 weeks):• Confirmation of the key uncertainty drivers and data sheet preparation, ECI data• Development of the risk schedule including integrity reviews and benchmarking• Development of the capital cost risk model structure and integration with the schedule model.
Reviewing the estimate benchmarking• Reviewing the project, procurement and contract risk registers• Planning the workshop agendas to capture the range data
• Schedule and capital cost workshops – 1 week
• Analysis, validation and results review with the project team. The review also included reviewing the QRA and parametric model results – 4 days
• Risk treatment and associated workshops – 1 week
• Final analysis including sensitivity analysis, a what-if analysis at a significant interface milestone and interaction with the ramp-up risk analysis – 3 weeks
• Reporting – 1 week
QRA MODELS – SCHEDULE RISK MANAGEMENTThe schedule risk management process follows the standard risk management process – to be effective all phases need to be completed and it is an iterative process.
KEY TAKE AWAY – need to ensure the integrity of the model and also have a shared understanding of
where it benchmarks. Rarely is enough time spent in the treatment phase of the process.
Establish the Context Identify Risks Treat RisksEvaluate RisksAnalyse Risk
Communicate and Consult
Monitor and Review
Develop the
Deterministic Schedule
Inputs:
- Execution Strategy
- Contract Packages
- Discipline input
- Schedule Basis
Processes and Outputs:
- Schedule Reviews
- Pertmaster reports
- Fuse Analysis
- Discipline input
- Schedule Basis
- Agreed schedule
Schedule Risk Assessment
(quantify the risk we must add to the schedule)
Inputs:
- Execution schedule (CP and near CP activities) vs Summary Risk Schedule
- Project Risk Register
- Workshop Briefing Note
- Agreed workshop participants and location
Processes and Outputs:
- Schedule ranging and risk quantification workshop – activity uncertainty & risks
- Schedule risk analysis using workshop data
- Review of risk profiles for Milestones that are being monitored
- Assess the schedule contingency and schedule accuracy – are they within
expectations?
- Review of the “key drivers” of the risk profiles -
- Evaluation of profiles – tolerable? And have we achieved ALARP?
- Are risk treatments required to improve the profile
Treat Risks
(de-risk & optimise)
Inputs:
- Risk Register
- Analysis profiles
- Tornado charts
- KPI’s
Processes and Outputs:
- Prioritise efforts using
the key drivers
- Develop treatments &
associated costs
- Test treatments – analyse
- Cost / benefit decision
- Finalise and agree final
schedule with stakeholders
Monitor and Review – Have our risks or controls changed? – are we implementing our risk treatments as per our plan?
Communication and Consultation – have all stakeholders been consulted, have we meet their needs, objectives and tolerability criteria?
QRA MODELS – COST RISK MANAGEMENTThe cost risk management process also follows the standard risk management process – to be effective all phases need to be completed and it is an iterative process.
KEY TAKE AWAY – rarely is enough time spent in the treatment phase of the process.
Establish the Context Identify Risks Treat RisksEvaluate RisksAnalyse Risk
Communicate and Consult
Monitor and Review
Develop the
Deterministic Estimate
Inputs:
- Execution Strategy
- Contract Packages
- Discipline input
- Estimate Basis
Processes and Outputs:
- Estimate Reviews
- Discipline input
- Estimate Basis
- Agreed estimate
Monitor and Review – Have our risks or controls changed? – are we implementing our risk treatments as per our plan?
Communication and Consultation – have all stakeholders been consulted, have we meet their needs, objectives and tolerability criteria?
Capital Risk Assessment
(quantify the risk that must be added to the estimate)
Inputs:
- Execution Estimate – Capital Risk Model Structure developed – based on key
drivers
- Project Risk Register
- Agreed workshop participants and location and Workshop briefing note
Processes and Outputs:
- Estimate ranging and risk quantification workshop – uncertainty & risks
- Capital risk analysis using workshop data
- Review of capital risk profiles
- Assess the capital contingency and accuracy – are they within expectations?
- Review of the “key drivers” of the risk profiles
- Evaluation of profiles – tolerable? And have we achieved ALARP?
- Are risk treatments required to improve the profile
Treat Risks
(de-risk & optimise)
Inputs:
- Risk Register
- Analysis profiles
- Tornado charts
- KPI’s
Processes and Outputs:
- Prioritise efforts using
the key drivers
- Develop treatments &
associated costs
- Test treatments – analyse
- Cost / benefit decision
- Finalise and agree final
estimate with stakeholders
QRA MODELS – SCHEDULE RISK MODELSchedule risk models are typically based on a risk schedule and a hybrid modelling approach adopted using both specific task ranging and risk factors. Risk events are inserted and time variable costs also included through tasks and/or hammocks.
QRA MODELS – SCHEDULE RISK MANAGEMENTSchedules are typically complex by their time basis and the logic that exists between numerous activities. Risk sources are also numerous and can impact many activities at the same time.
• Fundamental uncertainty in the work
• Productivity uncertainty, labour hours
• Maturity of the design and planning – material quantity uncertainties
• Unrealistic baseline schedule eg rates of progress
• Natural - weather, geological uncertainties, wave climate etc
• Project complexity
• Scheduling abuses
• Relying on participants outside the organisation
• Subcontractor is late
• Key construction equipment is not available as planned
• Quality of resources, management and supervision
• Direct and secondary impacts of change – design or construction
• Manufacturing or fabrication problems
• Government approvals and permits are late
• Contracting problems – market issues
• Supply chain and logistics
QRA RISK MODELS – BETING IT ALL TOGETHER AND PLANNING FOR YOUR INPUTSHaving established an agreed framework, it is vital to adequately prepare for collecting the required inputs to the risk model and how this data and information will be collected – typically in facilitated workshops.
Schedule
Scope- WBS- Contract Packages- Quantities, Resources, Productivity
Cost
Integrated Schedule and Cost Range
Analysis
Overall Contingency (P10, Mean, P90) by Delivery PackageTornado ChartsProbability DistributionsAccuracy of the Estimate / Schedule
Range Analysis Workshops- Min, Low, Mid, High, Max Range Assignment- Correlations- Datasheets
Uncertainties- supply prices- quantities- productivity- labour rates
Project Risk Register Risks- applicability- likelihood- impact on cost and schedule
Integration
Integration
Outputs
QRA MODELS – KEY ISSUES – UNDERSTANDING UNCERTAINTYThe time spent to understand the key uncertainties and how they should be modelled will assist the stakeholders to better understand the model and to test the sensitivity of these critical variables.
Projects basically consist of a scope, design and then the 3 M’s – men (or labour), materials and Machinery (or equipment). All have uncertainties and relationships with other variables that need to be modelled and taking labour cost uncertainty as an example:
Labour cost risk = f(labour base hours × available hours productivity × workface productivity × labour rate × material qty
× schedule duration)
QRA MODELS – KEY ISSUES – UNDERSTANDING UNCERTAINTYThe time spent to understand the key uncertainties and how they should be modelled will assist the stakeholders to better understand the model and to test the sensitivity of these critical variables.
FUNDING RISK
SCHEDULE RISK
TECHNICAL RISK
COST RISK
Productivity uncertaintyArea/Facility availability
Personnel availabilityEquipment/Material availability
Adverse environmental conditions
Funding constraintsPrioritization uncertainty
Under funding potential
Technology maturityPerformance requirements severity
Design data availabilityTest failure potential
Rework potentialProcess capacity adequacy
System reliability concernsSafety concerns
Escalation sensitivityLabor rate uncertainty
Equip & material $ uncertaintyEstimate completeness
QRA MODELS – KEY ISSUES – RANGING AND BIASESYou need to be part Psychologist to navigate your way through the risk management process. Human factors and biases need to identified and managed as best you can.
1. MOTIVATIONAL BIAS
Expert’s answers do not reflect beliefs
• Caused by personal involvement – what will my boss think, project won’t look good, risk mgt is too hard etc ,
• Too close to action
2. COGNITIVE BIAS
Expert’s answers do not reflect information, inexperience
• Anchoring – focus on a particular number
• Short term memory – focus on most recent event or data
• Long term memory – stuck on past events
• Hidden assumptions; interpretations taken as facts
• Overconfidence
Overwhelming evidence reveals that assessors probability density functions are too narrow & research
shows that 20 to 50% of true values lie outside the 0.01 and 0.99 fractiles, instead of the prescribed 2%
QRA MODELS – KEY ISSUES – RANGING AND BIASESYou need to be part Psychologist to navigate your way through the risk management process. Human factors and biases need to identified and managed as best you can.
QRA MODELS – KEY ISSUES – RANGING AND BIASES – TECHNIQUES TO ADDRESS BIASESThe primary method is the use of an influence or contributing factor chart from which scenarios can be developed and quantified.
QRA MODELS – KEY ISSUES – RANGING AND BIASESYou need to be part Psychologist to navigate your way through the risk management process. Human factors and biases need to identified and managed as best you can.
▪ Min and Max scenarios are firstly derived
▪ Low and High cases must be plausible
▪ A clear set of circumstances should be developed that describe each situation – clarity of the scenarios is important
Comfort Zone
HighLow
QRA MODELS – KEY ISSUES – OUTPUTS – VALIDATION AND SENSITIVITYDo the outputs make sense and explain what we believe about the real world of the project? Do the results respond as expected when sensitivity analysis is done? Does it help the team make decisions?
650 700 750 800 850 900 950 1000 1050 1100
Millions
Estimate$723.8M
P5 = $752M P50 = $823M P95 = $907M-8.6% +10.2%
P90 = $887.3M
Mean = $825.6M
P50 Contingency= $99.2M
Mean Contingency= $101.8M
P90 Contingency= $163.5M
86 124
18/11/2015
05/06/2016
Distribution (start of interval)
0.0
5.0
10.0
15.0
20.0
25.0
30.0
35.0
40.0
45.0
50.0
H
its
0% 19/08/2015
10% 02/10/2015
20% 23/10/2015
30% 06/11/2015
40% 24/11/2015
50% 17/12/2015
60% 18/01/2016
70% 11/02/2016
80% 11/03/2016
90% 19/04/2016
100% 06/10/2016
Cu
mu
lati
ve F
req
uen
cy
Coal Export Terminal - SRA - April 2011A1000.6 - First Coal + Global Project Risks : Finish Date = 22/09/2015
Mean = 2/1/2016
QRA MODELS – KEY ISSUES – OUTPUTS – VALIDATION AND SENSITIVITYDo the outputs make sense and explain what we believe about the real world of the project? Do the results respond as expected when sensitivity analysis is done? Does it help the team make decisions?
CP0001
CA2001
CG1601b
CC2002
CC0001
CE2801
CG2801
CR3101
CR3103
CG1102
CC2001
CC2003
CG1601
CG1101
CC3101
CC0003
CC2701
CG2001
CG2101
CG2102
CONTRACTS MEAN AND P90 PROFILE ($M’s )
P90 CONT MEAN CONT
Equipment - Mining
EPCM
Equipment - Other
Common Indirects
Construction Contracts
ESTIMATE CATEGORIES MEAN AND P90 PROFILE ($’M’s)
P90 CONT MEAN CONT
This risk exposure will require management focus
Bulks Pricing
Discrete Risks
EPCM
Schedule Driven - EPCM
Labour Rate
Owners Costs
PST
Schedule Driven - Escalation
Escalation
Equipment
Winter Weather Productivity
Common Indirects
Schedule Driven - Owners
Labour Productivity
Quantities / Scope Driven
Schedule Driven - Common Indirects
Schedule cost risk
Estimate cost risk
Discrete cost risk
PARAMETRIC MODELS – ANOTHER WAY TO VIEW THE RISK PROFILESParametric models are quite useful as they focus on the systemic project issues including the level of definition achieved based on the AACE classifications and are based on empirical analysis of project data.
Reference:
Project Risk Quantification
John K. Hollmann
The above method can also be adopted with factors for
distressed projects introducing non-linear impacts
PARAMETRIC MODELS – ANOTHER WAY TO VIEW THE RISK PROFILESThe results of the parametric model can be included in the usual QRA schedule and capital cost risk as a along with the other major risks and the analysis completed as usual.
Schedule risk profile = systemic risk + critical risks
Capex risk profile = systemic risk + critical risks + schedule risk* × burn rate
*Burn rate not applied to the systemic schedule risk.
The method has been used on a number of projects now with the results
from one megaproject analysis summarised below. The results in general
have been well aligned with QRA results
Risk Model Mean
Contingency
%
P10
Accuracy%
P90
Accuracy%
QRA 15% -12% +13%
Parametric 13% -15% +19%
Risk Model Mean
Contingency
%
P10
Accuracy%
P90
Accuracy%
QRA 13% -6% +7%
Parametric 8% -3% +4%
PARAMETRIC MODELS – SOME BENEFITSAs another process or approach in the QRA toolbox it provides some valuable insight into the risk profile especially in terms of the “soft” issues that are actually the “hard” elements to get right for projects.
• Relatively ease to apply
• Empirical basis – Hackney, Rand, IPA and Hollmann
• A good reality check on the potential impact of “soft” elements – e.g. team
development, quality of project controls
• View on execution complexity
• Can use in hybrid QRA Models
• Another view on the risk profiles
• Can develop your own in-house model based on your own data and
project delivery models
QRA MODELS – THE FUTURE
As digital delivery models and AI initiatives accelerate risk management and quantitative analysis will most likely become the norm and be done in real time. Some developments to follow include:
• BASIS – new company formed by Dan Patterson – knowledge based
planning using AI feedback in real time to build the plan
• BIM – integration of risk information in the various modules and real time
updates
QRA MODELS – THE FUTURE
As digital delivery models and AI initiatives accelerate risk management and quantitative analysis will most likely become the norm and be done in real time. Some developments to follow include:
• Development of more advanced risk algorithms for projects – current
research
• Real time feedback from the field for productivity and other key KPI’s and
integration with risk models for decision making
• Development of project simulation models - work by FICO for Shell on
their operations
• Others????????
MEET YOUR NEW RISK MANAGER – THE FUTURE HAS ARRIVED
