development and use of the ocm 1 © university of wisconsin-madison

© University of Wisconsin-Madison 1

Development and Use of the OCM

© University of Wisconsin-Madison 2

Development and Use of the OCM

3C 1999-2013, David H. Gustafson and Harold J. Steudel

Learning Objectives

Upon completing this lesson, you will be able to:• Understand the theoretical nature and

development of the Organizational Change Manager (OCM)

• Understand your results from the OCM and how to use them to improve your likelihood of project success

• Appreciate some level of confidence in the results you will get on your project


What is the OCM forecasting model &

how was it developed?


Identify Factors & Measures

• Interview each expert– You must predict projects’ success or failure.

– Can only talk to me. I will get answers.

– What questions before predicting?

– What answers would make you happy/sad?

• Review literature to supplement interviews.• Create “straw model”, and meet and fight till

agree.


Likelihood Ratio Estimation

• Estimation - Experts– Estimate likelihoods– Estimate likelihood ratios– Compare responses.– Modify likelihood estimates

• Compare & discuss differences across experts

• Re-estimate likelihoods

8C 1999-2013, David H. Gustafson and

Harold J. Steudel

Example Likelihood Estimation

• Direct Estimate: Suppose you had the records of 100 successful change projects. How many would have project launches “mandates” that are: – Proactive, – Neglectful and – Nonexistent (against).

0

10

20

30

40

50

60

Proactive Neglect Against

Proactive 35 10

Neglect 60 55

Against 5 35

Success Failure


Proactive Leaders carefully thought about this, Assigned a change agent (Champion), Gave a very clear aim for the project, Made not changing unacceptable and set a firm deadline . NeglectfulLeaders initiated the project, Assigned the change agent, Didn't clearly define need, task or deadlines Nonexistent (Against) Leaders were against project from start

The “Project Launch” Factor


Likelihood ratio estimation

• I have the records of two projects out in the hall. One is was a success; one a failure. In which one were senior leaders more likely to:

· carefully think about the project before picking it, · set a very clear aim,· remove status quo as an option and · give a champion needed responsibility & authority.

• How much more likely? Much? Some? A LITTLE! Almost even?


Example: Internal Comparison

• You said that a success would be a “little more likely” to have a proactive mandate.

• But your direct estimates say that a proactive mandate is

3.5 times more likely. A rather big difference.

• Please resolve this.


Compare Across Experts

• Look for substantial differences

• Ask them to talk about the differences

• Give them the opportunity to revise.

• Use the average their likelihood estimates

13C 1999-2013, David H. Gustafson and

Harold J. Steudel

Example: Internal Comparison

0

10

20

30

40

50

60

70

80

Proactive Neglect Against

Proactive 35 24 10 10

Neglect 60 71 55 65

Against 5 5 35 25

Success1 Success2 Failure1 Failure2


OCM Values for Project Launch Factor

Number of “Yes” OCM Factor Value

0 0.201 0.71

2 1.10

3 1.70

4 2.40

Bayesian Factors…….

Factor Desired Situation Number of Checks 4 3 2 1 0

Project Launch

Senior leaders:o carefully thought about project and then decided to do it,Ÿ assigned a champion to make the project succeed,o provided a very clear aim for the project,o made not changing unacceptable and set a firm deadline.

2.41

1.71

1.11

11.4

15

Senior leadergoals,

involvementand support

o Champion ensures project will help meet leader goalsŸ Champion makes sure leaders are informed & involvedŸ Leaders endorse the solutiono Leaders spend their time & resources to support project

. .

Middlemanager goals,involvementand support

Ÿ Champion makes project help meet middle manager goalsŸ Champion keeps middle managers involved & informedŸ Middle managers endorse the solutionŸ Managers spend time & resources to support project

. .. .

1/6 * 1/1.4 * etc. for the 15 factors


What exactly does it predict?

What is Success?

Six months after changes are made: they will still be in place and

those affected will (for the most part) say: “It worked. I am glad we made the change”

OCM Testing and Validation


Study I

• Fourteen nursing homes

• Team watched state surveyors: scored OCM

• Six months later returned; measured change

• Correlated scores with # deficiencies fixed.

• Correlation was .80.


Study II - Method

• 323 senior health care leaders: US, Canada and Netherlands (e.g. administrators, medical directors) attending professional meetings.

• Completed a survey: project they know well.

• 194 implemented >6 months & results known.

(Big Success, Modest Success, Modest Failure, Big Failure)


Assessed predictive accuracy

• Divide outcomes in two groups Big Success or Pretty Successful Disappointing or Big Failure

• ROC analysis chose three cutoff regions < -1.0 (29% of cases) -1.0- +1.0 ( 8%) >+1.0 (63%)

• Success & failure rates in each category?


0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150

<-1

Between -1 to +1

>+1

Number of cases

Sc

ore

Failure

Success

Accuracy of Change Predictor


Accuracy of OCM

• 66 failures: OCM predicted 77% accurately

• 151 successes: predicted 88% accurately

C 1999-2010, David H. Gustafson and Harold J. Steudel

22

The OCM “Predictor” Score portrays your chances of success.

For instance, a +6 score means that projects with this score succeed 6 times more frequently than they fail. And a score of -8 means that projects with this score fail 8 times more frequently than they succeed.

Prediction of Propensity for Successful Change

Based on the Team’s responses, your organization scored

-2.646 (Process) and -3.85 (Cultural) on a scale from –10 to +10 for this project.

0.000

1.000

2.000

3.000

4.000

5.000

6.000

7.000

8.000

9.000

OCM Factor

Potential vs. Actual Predictor Scores

Potential

Actual

Prediction of Propensity for Successful Change

OCM Score vs. Success & Failure Rates

0

5

10

15

20

25

30

35

-3 to -5 -1 to -3 -1 to +1 +1 to +3 +3 to +5 +5 to +8 +8 to +10

Success Failure


26

We find that scores in the +9 to +10 range rarely occur in the “real world” and probably indicate that the team doing this scoring has not been brutally honest with themselves.

Special Note


27

Key Benefits

• Helps teams identify & correct roadblocks

• Helps monitor progress of change effort

• Helps select changes to address

• Helps allocate implementation resources

• A metric for identifying positive & negative implementation patterns in organization

Interpretation of YOUR Overall OCM Scores

• For OCM scores over 3– If your team has OCM Predictor score over 3, your team

does not need to make any immediate improvements to increase your likelihood of having a successful project. However, the improvement could be made. These improvements can be considered “optional” at this time, provided the currently strong OCM factors do not deteriorate.

• For OCM scores under 3– If your team has OCM Predictor score under 3, your team

should consider if it is possible to make some improvements in order to increase your likelihood of having a successful project.

How to Gain the Opportunities?

How Can YOU Use the OCM Results to Improve Your Team’s Chances of Success?

development and use of the ocm 1 © university of wisconsin-madison

Documents

ocm forecasting model

odds of rain

experts5the ocm model

ocm calculator

terms of prior odds

key elements of prior

pd1h1pd1h2 posterior

forecasting model thats