27 ISMOR

Chris Tilney, Atkins Ltd2 September 2010

Learning from Experience (LFE) - MCA, SWIFT & HAZOP

Structure of Presentation

● Multi Criteria Analysis (MCA)

– Lessons learned from decision support applications in defence

● Structured What If Technique (SWIFT) & Hazard & Operability (HAZOP)

– Brief introduction to the techniques

– Lessons learned from recent nuclear safety applications

– Potential non safety related applications in defence

Multi Criteria Analysis (MCA)

MCA – LFE (1)

● Context for lessons learned

– Primarily focused on application of MCA to UK MoD procurement competitions i.e. down select decisions on Pre Qualification Questionnaire (PQQ) responses and Tenders

– Personal opinions of presenter, not necessarily shared by other Atkins staff or MoD personnel!

MCA – LFE (2)

● Structure of Assessment Hierarchies

● Derivation of Weighting Factors

● Preparing the Evaluation Team

● Scoring Criteria

● Aggregation

● Audit Trail/Toolset

● Presentation of Results

MCA – LFE (3) Structure of Assessment Hierarchies

Tower 1 Tower 2 Tower 3

Merits of solution offered

Capability to deliver

solution

Commercial Aspects

Price

Terms & Conditions

(T&Cs)

Performance

System Requirements

Document (SRD)

Compliance

People

Processes

Tools

Plans

Delivery Schedule

● Senior Decision Makers/Scrutiny favour:– Towers not weighted relative to each other

– No single, overall Figure Of Merit (FOM) score

● Small set of criteria (15 – 20) works best

MCA – LFE (4)

● Derivation of Weighting Factors

– Pairwise comparison works best. Three ‘rounds’ to derive weights:● 1st round: initial pairwise scores followed by explanation of outliers

● 2nd round: score again then calculate weights based on consensus scores

● 3rd round: review weights, adjust if necessary (with rationale)

– Group dynamics, watch out for/control:● Dominant personalities

● Deferral to rank (if military Subject Matter Experts (SMEs))

● Non assertive gurus

● Preparing the Evaluation Team

– Evaluation Guide: helps enforce coherency, transparency, robustness of scoring process

– Kick-off group brief: ditto

– 1-2-1s on toolset: reduce risk of failing to keep to evaluation schedule

MCA – LFE (5) ● Scoring Criteria

– Scoring scale● Absolute

● 0 to 10 point scale

● 3 reference points: 10, 7 & 4 with guidance for assessors

● No ‘extra’ marks for exceeding requirement

– Confidence adjusted scores (demonstrable/evidence based capability)

– Minimum of 2 Assessors: Lead & Shadow(s)

– Moderators: resolve disagreements, holistic view/input to assessors

● Aggregation

– Automate at lower levels of hierarchy

– “Man-in-loop” at higher levels of hierarchy

– Trigger points/capability thresholds for all criteria that are scored

MCA – LFE (6) ● Audit Trail/Toolset - AWARD

– Simultaneous, distributed input (over Restricted LAN Interconnect (RLI))

– Strong functionality for:● Capturing assessor rationales, issues & concerns

● Monitoring progress of evaluation

● Reports/interrogation

● Presentation of Results

– KISS principle to avoid information overload for Assessment Panel Members

– Executive Summaries of what is being offered

– Bar graphs for Towers 1 & 2

– ‘Horse blanket’ for Tower 3 T&Cs

– Risk adjusted Whole Life Cost (WLC) plots for price

– Combined Operational Effectiveness & Investment Appraisal (COEIA) plots for Value for Money (VfM)

– Slides on major concerns, issues and ‘red flags’

MCA – LFE (7) ● Commercial Issues

– Central finding of recent court judgements (e.g. case “Lettings International Ltd v London Borough of Newham”):Any aspect of the Authority’s requirements or the way that it will evaluate tenders that could affect the way a bidder prepares its bid must be fully disclosed before tenders are returned.

– Implications may include, for example:● Assessment hierarchies – fully weighted, automatic roll up of weighted scores

● Disclosure of assessor scoring guidance to tenderers

● Predefined, prescriptive COEIA methodology (i.e. can’t be tailored after tenders received). Dstl study is investigating this issue.

– Are we entering dangerous waters?

i.e. OA/Decision Support Analysis no longer informing decision makers but effectively making the decision (by option scores and option ranking) by slavish adherence to a detailed, predefined process

MCA – LFE (8)

● Overall LFE findings

– Well understood/accepted by Senior Decision Makers

– Works well with careful preparation and checks & balances on aggregation

– BUT

– Can be very resource intensive

– Commercial issues may prevent/limit future application of some of the LFE and undermine central tenet of OA to inform decision makers

Structured What If Technique (SWIFT) & Hazard & Operability (HAZOP) – Intro to techniques

SWIFT – Intro to Technique (1)

● Structured brainstorming method for analysing process or system

● Usually applied to systems or processes not deemed to be safety critical but which do have safety related failure modes

● Approach– Select system, subsystem or process

– Chair poses pre-planned ‘What if’ questions, identified via● Task analysis

● Basis of Design

● Generic Checklists

● Process Description

● Standards, regulations & guidelines

● Past incidents & accidents

– Multi disciplinary team (design, operation & maintenance) of SMEs answer ‘What if’ questions

– Results captured in log sheet by scribe

SWIFT – Intro to Technique (2) Example SWIFT Log sheet

Project: Aircraft Hangar Page No

Description: Aircraft movements inside and outside the Hangar space

Question Category

What If Consequences Safeguards Record No

Recommendations

External Factors and Influences

High Winds Potential collision between Vehicle and Hanger side walls during operation

Handling crew properly trained. Vehicle only operated under strict weather conditions. Weather reports regularly made during operations

RN1 Handling crew training to include regular practice of scenario

Operating errors and other human factors

Handling operator hitches up vehicle incorrectly

Vehicle breaks free of handling crew control during operation. Collision could result in injury to personnel and damage to vehicle / superstructure

Handling crew properly trained. Vehicle parking brakes manned during operation

RN2 Include double checks when handling vehicle

Handling operator directs vehicle into obstacle

Vehicle and or superstructure damaged, personnel injured through collision, crushing

Handling Crew Properly trained. Operator fully supported by crew when handling the vehicle

RN3 Pre operation brief to be given to Handling crew.

Additional trained personnel to be made available during operation.

Step 1

Step 2

HAZOP – Intro to Technique (1)

● Considers the operation (operability) of a system or subsystem

● Systematically identify deviations from the design intent which could lead to Hazards or operability problems

● Multi-disciplinary SME team activity

● Makes recommendations which could influence the design

● Approach– Use Guide Word / attribute combination to suggest possible deviation(s)

– Discuss and agree all credible causes

– Identify all consequences, identifying Hazards

– List existing safeguards

– Discuss and agree further actions

– Record

– Repeat until complete

HAZOP – Intro to Technique (2)

Project Vehicle Fuel System Page No

Description The fuel tank, engine inlet/outlet pipes and hoses, fuel pump, fuel filter

Guideword Causes Consequences Safeguards Record number

Recommendations

More Flow Fuel pump out of calibration

Engine floods and stalls, fuel leaks from system and pools under vehicle with the potential to ignite upon contact with hot surfaces

Operators trained to check and maintain fuel pump at regular intervals

RN1 Fit fire extinguishers in vehicle cabins. Provide training to check for leaks when vehicle stalls

Reverse Flow Fuel pump incorrectly set up during fit

Major spillage of fuel with potential to ignite and cause major fire

Replacement fuel pumps are pre use checked before dispatch from manufacturers

RN2 Fit Non return valve in fuel system

Backflow experience during a pressure drop caused by fuel pump failure

Engine cuts out. Small fuel spillage may be experienced with potential to ignite

Fuel pumps are checked and maintained by operators and maintainers at set intervals

RN3 Fit Non return valve in fuel system

Example HAZOP Log sheet

SWIFT & HAZOP - LFE

SWIFT & HAZOP – LFE (1)

● Preparation– Key to effective workshop

– Don’t circulate ‘what if’ Qs in advance to help avoid preformed views

– Ensure reference material to-hand during workshop e.g. large-scale design drawings

● Attendees– Number: No more than 8, including Chair & Scribe

– Expertise: Full coverage required (tension with number!)

– Chair:● Strong/assertive

● Independent of the project

● Ideally knows attendees to draw them out, but not too well (bias/favouritism)

● Have sufficient technical knowledge to understand the discussion and be able to apply judgement when directing the study, e.g. distinguishing between superficial/trivial issues and important issues

– Scribe: Understands domain and technical jargon

SWIFT & HAZOP – LFE (2)

● SWIFT vs HAZOP– SWIFT

● Don’t have to have finished design i.e. can apply to concepts

● Quick and easy to implement (cf HAZOP)

– HAZOP● Mature design needed

● Exhaustive analysis, completeness of analysis

● Time consuming/resource intensive

SWIFT & HAZOP – Potential Non Safety Applications

SWIFT & HAZOP – Potential Non Safety Applications

● Explore/define CONEMP

● Develop OA scenarios & vignettes

● Explore and assess merits of business change programme

● General risk identification & analysis