Exploitation of OA techniques to support IA & decision making

9th Feb 2010Colin Drysdale

E-mail : colin.drysdale@atkinsglobal.com

2

Overview

• What is Operational Analysis (OA)?

• Difficulties facing a decision maker – how can the analytical community help?

• Ways to inform a decision maker

• Ways to ensure the decision maker understands the information

• An example of a technique that sets out to both inform & make the information understandable

3

What is Operational Analysis (OA)?

• OA is the name given by the military to Operational Research (OR)

• OR, also known as Operations Research or Management Science (OR/MS) is the discipline of applying advanced analytical methods to help make better decisions– Source: The OR society (http://www.orsoc.org.uk)

4

Typical outputs of OA

• The articulation of analytical problems to be solved

• Identification of the solution space

• Mathematical theories

• Suggested policies that could optimise systems or satisfy goals

• Logical structuring of issues– Leading to insights

• Measures of Effectiveness– Rather than Measures of Performance

5

Difficulties facing decision makers

6

Difficulties facing decision makers

• The influence of “providers” over “deciders”

• The maturity & availability of solutions

• Irreconcilable starting positions

• Multiple goals & constraints

• Uncertainty & risks

• Trade-offs

• Funds

(Not an exhaustive list)

7

What makes decisions succeed or fail?Informed

Investment Decisionmade

Decision makerinformed

Decision makerheads info'

Rigorous,uncertainty bounded

"true" cost info'for all options

Rigorous,uncertainty bounded

operationaleffectiveness info'

for all options

Assumptions arevalid (fit for

purpose)Method forcombining

performance data inscenarios is fit for

purpose

Scenarios are fitfor purpose Performance data

are fit for purpose

Cost estimatesare fit for purpose

Method forcombining cost

estimates inscenarios is fit for

purpose

&

&

& &

&&&

&&

&&

&

Decision makerunderstands info'

&

8

How can the analytical community help decision makers?

• We can calculate and collate information to inform decision makers

– Inform them of the cost implications of investment options

– Inform them of the operational effectiveness impact of the options

9

How can we help decision makers understand the gathered information?

• One way to inform decision makers that, additionally, seeks to help decision makers understand the gathered information is called Multi Criteria Decision Analysis– An example is coming up shortly

• Another way is to present information in more than one format – E.g. both tables & graphs

• Yet another way is to try to turn the implications of numbers back into words for the decision maker

10

Introducing MCDA as one way to inform decision makers

11

Background to MCDA

What MDCA is:• MCDA is the name given to a group of techniques by the OR Society

• A technique for supporting decision makers who are trying to simultaneously satisfy multiple, potentially conflicting goals

• MCDA techniques pop-up in all sorts of domains with different names & conducted in different ways to different standards of analytical rigour

• MCDA is a technique for helping decision makers make choices– Arguably it does not:

• do predictive modelling• produce Measures of Effectiveness

12

One way to inform decision makers

• It is difficult to view MCDA as an “advanced” analytical technique– It is classed as a form of Soft OA

• MCDA requires little or no mathematical knowledge to use– But beware of this

• Like any technique it has pit falls, these can be avoided by good practitioners, who understand the features of the decision to be supported

• Decision making Culture• Procedures• Standards of analytical rigour

13

What does MCDA do?

• Produces a Figure of Merit (FOM)

• Combines disparate criteria or factors

• Provides the relative ranking of the options

W2

W1

W3

Sa

FOMSb

Sc

FOM = Σ Si . Wi

14

MCDA sequence of events

1. Network of criteria / requirements

2. Criteria that can be measured, predicted, estimated, judged, or populated from available data

3. Define the relationship between performance and worth

4. Weight the links within the network

5. Evaluated criteria at nodes on the network (& then calculate the scores)

6. Conduct sensitivity analysis

7. Review findings with decision maker

15

MCDA Example

16

How MCDA works – simple example

• Equipment investment decision

• Equipment - Bicycle front light

• Scenario 1

– 10 mile daily commute after dark on a mixture of lit town roads and un-lit dual carriage way, single carriageway, & single track roads with passing places

• Requirements:

• Vision - to enable no reduction in speed from day light conditions

• Presence - no reduction in recognition to other road users compared to day light conditions

• All weather capable

• Light weight

• Reliable - very low probability of failure in use

• Reliable - graceful degradation or failure warning

• Available – for expected maximum mission duration

• User able to choose where to fix the light

Warning! – this simplified example contains some elements notappropriate to MOD Business Case support

17

14 Vision

15 Presence

17 Fixing options

18 Availability

21 FOM Scenario 1

22 Purchase Cost

23 Running Costs

36 weight

37 All Weather

1. MCDA network of criteria

18

14 Vision

15 Presence

17 Fixing options

18 Availability

21 FOM Scenario 1

22 Purchase Cost

23 Running Costs

24 Oversize

26 Helmet

27 Head

28 Luminous flux

31 Failure warning

33 Mission lengthsupported

35 Flashing mode

36 weight

37 All Weather

1. MCDA network of criteria

WARNING: This is not necessarily howyou should combine cost information

for Government Investment decisions consult the Treasury “Green Book”.

19

2. Criteria that can be “measured”

Lm

(£)

(£ per annum)

Y/N

Y/N

Y/N

Hr

Y/N

g

Y/N

14 Vision

15 Presence

17 Fixing options

18 Availability

21 FOM Scenario 1

22 Purchase Cost

23 Running Costs

24 Oversize

26 Helmet

27 Head

28 Luminous flux

31 Failure warning

33 Mission lengthsupported

35 Flashing mode

36 weight

37 All Weather

No reliable information

Measured Quantity(measurand i/p)

Yes/No Question(Boolean i/p)

Subjective SMEjudgement

20

3. Performance & worth relationships

Lumens

Burn Time

Weight

500 (g)

(hr)10 2 3

1,000100

Value

Value

Value

Measured Quantity(measurand i/p)

Score

(0≤ Score ≤1)

Score Performance

1

0

1

0

1

21

3. Performance & worth relationships

PurchaseCost

(£)

RunningCost

(£)

100 200 30075

Value

Value

10 15 30

Measured Quantity(measurand i/p)

Score

(0≤ Score ≤1)

Score Performance

0

1

0

1

22

4. Weight contributions to outcomes - weight network links

14 Vision

15 Presence

17 Fixing options

18 Availability

21 FOM Scenario 1

22 Purchase Cost

23 Running Costs

24 Oversize

26 Helmet

27 Head

28 Luminous flux

31 Failure warning

33 Mission lengthsupported

35 Flashing mode

36 weight

37 All Weather

0.20.10.2

0.2

0.1

0.10.1

1

0.5

0.50.7

0.30.33

0.33

0.33

Contribution Weight

(ΣWi = 1)

Weight Contributions

23

Aspects of good practice

24

Sensitivity Analysis

• Having got initial results out of the models may have felt like arriving at the destination …

• This is NOT correct

– the Analysts job is not done

– there is a need to conduct Sensitivity Analysis

– to understand how uncertainty affects the ranking of the options

– indeed once uncertainty is considered it may not be possible to

discriminate between the options

25

5. Evaluate criteria – for each Option - Top FOM results

Option N Make N ModelPurchace (£)

Running (£ pa.)

OP (Lumens)

Weight (g) FOM

1 Dog ear Hedo 31.49 3.78 80 118 0.6413 Faith Four 1 254.49 0.51 960 420 0.5914 Faith Four 1+1 309.49 0.51 960 420 0.59

8 Chilly Minn Mum 291.49 0.51 960 320 0.5616 Heavy emotion Susan 200 79.99 0.51 200 240 0.54

26

Flaws with these answers

• Option 1, The Dog ear, “Hedo” should not come out as the best option as it is not really bright enough to be fit for purpose in this Scenario– It is successful because of its light weight and low purchase

price

Lumens

1,000100

Value

0

1

Lumens

1,000100

Value

0

1

27

Flaws with these answers

• Options 13 & 14, The Faith, “Four 1” and The “Four 1+1” should not be equal in merit

100 200 30075

PurchaseCost

(£)

Value

0

1

100 200 30075

PurchaseCost

(£)

Value

0

1

PurchaseCost

(£)

Value

0

1

28

7.Review findings with decision maker

• Should not use this technique in the absence of decision makers from relevant parts of the process

• Review findings with decision maker

• Sometimes the analysis (or thought process) does not get as far as MCDA results– We gain some insight that reframes or satisfactorily solves the

exam question before the analysis is concluded

29

What MCDA can include

• Scenarios

• Linear or non-linear conversion of performance against each criterion into some from of worth to the decision maker (e.g. military worth)

• Sensitivity analysis (e.g. to address uncertainty & risk)

30

How I actually made the decision

31

Faith, Four 1

Chilly, Devil

Faith, Two 1

Dog ear, Hedo

How I actually made the decision

PurchaseCost (£)

Lumens

Faith, Sight

0

100

200

300

400

500

600

700

800

900

1000

0 50 100 150 200 250 300

32

Conclusion

Previously: – Discussed the role of causal mapping to make the development of

MCDA networks adequately rigorous to support Business Case decisions

Today:– Stated IA and OA studies can inform decision makers– Considered failure modes for decisions & how the analytical

community can help decision makers succeed– Explained what MCDA is and what it does– Identified that there are good and bad practices– Suggested some issues for discussion covering which practices

are acceptable for any given purpose

33

Thank you for your time

Any Questions?

Colin Drysdale

E-mail : colin.drysdale@atkinsglobal.com

34

END

Colin Drysdale

E-mail : colin.drysdale@atkinsglobal.com

35

Spares

Colin Drysdale

E-mail : colin.drysdale@atkinsglobal.com

36

Options and data

37

Evaluated criteria – evaluate network “leaf” nodes

Option 1 2 3 4 5 6 7 8 9 10N Make Dog ear Dog ear Chilly Chilly Chilly Chilly Chilly Chilly Faith FaithN Model Hedo S4 Control Control Minn2Devil Mada Bull Minn Mum Sight Two 1Purchace (£) 31.49 269.99 148.49 162 201.49 219.49 247.49 291.49 72 164.49Running (£ pa.) 3.78 0.51 0.51 0.51 0.51 0.51 0.51 0.51 68.36 0.51Oversize 1 0 1 1 1Helment 1 1 1 1Head 1OP (Lumens) 80 700 240 240 700 480 700 960 240 480Fail warn 1 1 0 1 0 1 1 1Burn time (hr) 45 3 3 3 1 3 3 3 2.75 2Flash 1 0 1 1 1 1 1 1Weight (g) 118 497 98 98 102 228 276 320 220 275

Option 11 12 13 14 15 16 17 18 19 20N Make Faith Faith Faith Faith Heavy emotion Heavy emotion Hoff-mister Hoff-mister Hoff-mister Hoff-misterN Model Two 2 Two 1+1 Four 1 Four 1+1 Susan 120 Susan 200 Small Small 110 Small Small 110 +Small Small 100X2 Small Samll 400Purchace (£) 204.49 204.49 254.49 309.49 62.5 79.99 79.99 109.49 174.99 224.99Running (£ pa.) 0.51 0.51 0.51 0.51 0.51 0.51 0.51 0.51 0.51 0.51Oversize 1 1 1 1 1 1 1 1 1 1Helment 1 1 1 1 1Head 1 1 1OP (Lumens) 480 480 960 960 120 200 110 110 200 400Fail warn 1 1 1Burn time (hr) 4 2 2 2 2 2 3 3 3 2Flash 1 1 1 1 1Weight (g) 405 275 420 420 280 240 175 175 230 317

38

And the answer was ….

Option 1 2 3 4 5 6 7 8 9 10N Make Dog ear Dog ear Chilly Chilly Chilly Chilly Chilly Chilly Faith FaithN Model Hedo S4 Control Control Minn2Devil Mada Bull Minn Mum Sight Two 1Purchace (£) 31.49 269.99 148.49 162 201.49 219.49 247.49 291.49 72 164.49Running (£ pa.) 3.78 0.51 0.51 0.51 0.51 0.51 0.51 0.51 68.36 0.51Oversize 1 0 1 1 1Helment 1 1 1 1Head 1OP (Lumens) 80 700 240 240 700 480 700 960 240 480Fail warn 1 1 0 1 0 1 1 1Burn time (hr) 45 3 3 3 1 3 3 3 2.75 2Flash 1 0 1 1 1 1 1 1Weight (g) 118 497 98 98 102 228 276 320 220 275

FOM 0.64 0.40 0.49 0.53 0.51 0.46 0.51 0.56 0.50 0.50

Option 11 12 13 14 15 16 17 18 19 20N Make Faith Faith Faith Faith Heavy emotion Heavy emotion Hoff-mister Hoff-mister Hoff-mister Hoff-misterN Model Two 2 Two 1+1 Four 1 Four 1+1 Susan 120 Susan 200 Small Small 110 Small Small 110 +Small Small 100X2 Small Samll 400Purchace (£) 204.49 204.49 254.49 309.49 62.5 79.99 79.99 109.49 174.99 224.99Running (£ pa.) 0.51 0.51 0.51 0.51 0.51 0.51 0.51 0.51 0.51 0.51Oversize 1 1 1 1 1 1 1 1 1 1Helment 1 1 1 1 1Head 1 1 1OP (Lumens) 480 480 960 960 120 200 110 110 200 400Fail warn 1 1 1Burn time (hr) 4 2 2 2 2 2 3 3 3 2Flash 1 1 1 1 1Weight (g) 405 275 420 420 280 240 175 175 230 317

FOM 0.38 0.48 0.59 0.59 0.46 0.54 0.53 0.51 0.38 0.40

Because I was just generating an example I did not conduct any sensitivity analysis

39

4. Objective & Threshold definition

1.0

0

Value

Top Speed(miles per hour)

Top Speed

100 150 200

1.0

0

Value

Top Speed(miles per hour)

Top Speed

100 150 200

or “effectiveness envelope”