Analysis by Competing Hypotheses

A business tool from The Psychology of Intelligence Analysis by

Richards J. Heuer, Jr., CIA, 1978-86

adapted for business by John Braren, Jr.with example

Why this presentation?

To put some extra time to good use while between contracts, I decided to create several presentations of some tools I believe could be useful for business.

This is the first presentation. I have made this as brief as possible in an

attempt to not kill interest, but I have no doubt some points deserve more development. Please feel free to contact me at JBraren@nc.rr.com if you want to discuss this further, or to offer comments to improve the presentation and the clarity.

BI is probably wrong / FBDM

TMI- Too much information. Fact Based Decision Making can mean more “volume” and less “quality”.

Too many biases built in to collection

Programmed BI causes pre-filtering and predetermined hierarchies and answers

Limits data with collection mechanisms

Current Process is troublesome

Select solution and find “proof”

This can yield the wrong answer for all the right reasons

50:50 chance to get right answer - for all the wrong reasons

Two bad examples

Wrong decision, for right reasons Buy a copier that is cheap, saves $$ and ink, has small foot

print But – can‟t print from remote, can‟t queue job for off-

hours, and only holds 100 sheets so need to monitor all jobs

Right decisions, for wrong reasons Build a bridge: 1- let‟s make out of steel, it‟s nice and shiny 2- I don‟t like driving over water, so let‟s put at

narrow point of river 3- Make lowest point of bridge at least 76‟ over water;

the mast on my sailboat is 72‟

So why does „satisficing‟ and FBDM persist?

Habit and comfort◦ More comfortable with failure than change

We are surrounded by the practice of deciding and then developing CYA support◦ Rather than stretching to find the most possibilities

and then expending effort to disprove most of them

More lucrative to sell BI tools and code than a decision making skill

What is ACH?

Analysis by comparative hypotheses

Developed for the CIA in 1978 – 1986

Based on:◦ - finding most possible answers◦ - applying ALL pro/con data against ALL hypotheses

◦ - disproving possibilities, not „proving‟ selections

The steps 1 – 4 of 8

1. Identify all the possible hypotheses to be considered.

2. List all significant evidence and arguments for and against. Combine to one matrix – all evidence for all hypotheses.

3. Identify the evidence and arguments that were most diagnostic.

a) All + or all – of no decision making value

4. Refine the matrix definitions as needed-Hypothesis, Evidence, Original Question.

The steps 5 – 8 0f 8

5. Evaluate each hypothesis. Disprove hypotheses and eliminate, rather than prove them.

6. Find the lynchpin items of evidence. Scrutinize these.

e) The conflicting + and – decision points

7. Report the conclusions. Discuss the relative likelihood of all the hypotheses, not just the most likely one.

8. Identify milestones for future observation, to monitor and re-evaluate analysis conclusions.

Example: Safety vests

Initial variables for decision might be

◦ Colors

◦ Size

◦ Cost

Which is best vest to buy?

V1 V2 V3 V4 V5 V6

Color F Orange F Orange F Green F Green Black Tan

Size Small All All All All All

Cost $1 $2 $3 $4 $ .10 $ .05

V5 and V6 might seem best buys, but colors don‟t seem right.Need to add element of “Visibility”.

Which is best vest to buy? - 2

V1 V2 V3 V4 V5 V6

Color F Orng F Orng F Green F Green Black Tan

Size Small All All All All All

Cost $1 $2 $3 $4 $ .10 $ .05

Visibility 2 mile ½ mile ¼ mile 1 mile 1/100 mile

1/100 mile

**With Visibility added, we see color isn‟t a decision factor.**Small only size won‟t work for all our users, so eliminate V1. V5 and V6 are unacceptable distances, so eliminate these.**V3 is less visible for more money than V2; eliminate V3**And back to Visibility, what do we need? 1 mile for hunting season or ½ mile for traffic visibility? This is our Lynchpin data.

A Real Example

The worst answer for all the best reasons

This real example works through a brief version of the steps that went into making a less valuable decision.

99+% consensus was the first decision was best.

Where do we start a new business system?

The three options were:

US based established company plant which produces for largest (80% revenue) customer

Foreign established plant, no large customer production

Newly acquired US plant, no large customer production

The Evidence for hypotheses - #1

80% facilities Non-80% facilities New facilities, non-

80%

1. Can‟t afford to trouble

80% customer_ _ __ + + ++

2. Most of team from 80%

facility, want to avoid stressn/a _ +

3. Need to learn new

system, so might as well do

once

n/a n/a ++

15

This was the evidence used to make the actual decision. It seems to lead to an obvious conclusion (“new facility”) based on the positive data. This was the path that was followed.

ACH forces the search for the lynchpin evidence and guards against finding support for the obvious, which is too often wrong.

The Evidence for hypotheses - #2

80% facilities Non-80% facilities New facilities, non-80%

1. Can‟t afford to trouble 80%

customer_ _ _ _ + + ++

2. Most of team from 80%

facility, want to avoid stressn/a _ +

3. Need to learn new system,

so might as well do oncen/a n/a ++

4. Employees feel pain, want

new system+ + n/a

5. Employees need to keep

some of old, don‟t want new

system

n/a n/a _

6. Unique unfamiliar

measurement systemn/a n/a _

16

With the help of hindsight, I have added the last three elements of evidence to the matrix. If the search for evidence had been an active exercise to consider all stake-holders and the complete „state‟ of the implementation, these elements would have been discovered and considered from the start.

Evaluating Evidence through #2

1. The first hypothesis of starting with the 80% customer facility has a huge negative and can be eliminated with confidence. (Note that significance of evidence mat be very subjective. If there is any doubt, the hypothesis probably should be kept in play.)

2. If we look at evidence element 2 (team from 80% facility), we can see that the evidence might be re-stated as “Team from outside facility”, in which case it carries the same negative (or positive) weight for both remaining facility types.

3. And now elements 5 &6 add two negatives to the „new facility‟ hypothesis which makes this our next best choice to cut as an option.

4. But now we want to add a 7th piece of evidence.

17

The Evidence for hypothesis - #380% facilities Non-80% facilities New facilities, non-80%

1. Can‟t afford to trouble

80% customer_ _ _ + + ++

2. Most of team from 80%

facility, want to avoid stressn/a _ _

3. Need to learn new system,

so might as well do oncen/a n/a ++

4. Employees feel pain, want

new system+ + n/a

5. Employees need to keep

some of old, don‟t want new

system

n/a n/a _ _

6. Unique unfamiliar

measurement systemn/a n/a _

7. Rationalizations usually

add to scope, but:

Bringing facility into 80%

methods will build “corporate

team” perception

n/a + +

18

Review each hypothesis

Review validity of each hypothesis; eliminate as possible

Evaluating Evidence through #3◦ Evidence elements 2 & 7 have equal values so they can

be eliminated as not useful.

◦ Because we are trying to disprove hypotheses, the two negatives (elements 5 & 6) for “new facility” become the only two valuable elements. These are the lynchpins.

19

The Conclusion80% facilities Non-80% facilities New facilities, non-80%

1. Can‟t afford to trouble

80% customer_ _ _ + + ++

2. Most of team from 80%

facility, want to avoid stressn/a _ _

3. Need to learn new system,

so might as well do oncen/a n/a ++

4. Employees feel pain, want

new system+ + n/a

5. Employees need to keep

some of old, don‟t want new

system

n/a n/a _ _

6. Unique unfamiliar

measurement systemn/a n/a _

7. Rationalizations usually

add to scope, but:

Bringing facility into 80%

methods will build corporate

team sense

n/a + +

20

Conclusion note - Item 4

80% facilities Non-80% facilities New facilities, non-80%

4. Employees feel

pain, want new

system

+ + n/a

21

Note that while this point may seem important, the fact is that it was a non-starter from the beginning.** With 2 “reasons for” and an “n/a” it had no dis-prove value.** And even after eliminating the “80% facility”, it still had no dis-prove value for the last two choices.** This is a clear example where bias for a solution can be seen as contrary to making a best choice decision.

ACH Steps Review

1. Identify all the possible hypotheses to be considered.

2. List all significant evidence and arguments for and against. Combine to one matrix – all evidence for all hypotheses.

3. Identify the evidence and arguments that were most diagnostic.

4. Refine the matrix- Hypothesis, Evidence, Original Question.

5. Evaluate each hypothesis. Disprove hypotheses and eliminate, rather than prove them.

6. Find the lynchpin items of evidence. Scrutinize these.

7. Report the conclusions. Discuss the relative likelihood of all the hypotheses, not just the most likely one.

8. Identify milestones for future observation, to monitor and re-evaluate analysis conclusions.

ExerciseAs a group, or individually, apply the ACH process to a decision you might make, or best, to one decision that worked and one that did not work.

The comparison of the historical decisions might drive home the value of the ACH approach.

The exercise can be done quickly and still show its value: Identify question, refine, and follow through rest of

steps quickly Best done on flip chart or white board, or an Excel

grid. Whatever works.

The Evidence for using ACHSelect and then find

support

ACH Process

1. Find most possible

solutions or responses - ++

2. Disqualify options that do

not work - ++

3. Keep options until they

are disproved; scientific - ++

4. Refine the question, the

evidence, and the solution

throughout the process

- ++

5. Avoid bias - ++

24