Evaluating Evaluating Organizational Organizational Change: How and Change: How and Why?Why?

Dr Kate Mackenzie DaveyDr Kate Mackenzie DaveyOrganizational PsychologyOrganizational PsychologyBirkbeck, University of LondonBirkbeck, University of Londonk.mackenzie-davey@bbk.ac.ukk.mackenzie-davey@bbk.ac.uk

22

AimsAims

Examine the arguments for evaluating Examine the arguments for evaluating organizational changeorganizational change

Consider the limitations of evaluationConsider the limitations of evaluation Consider different methods for Consider different methods for

evaluationevaluation Consider difficulties of evaluation in Consider difficulties of evaluation in

practicepractice Consider costs and benefits in practiceConsider costs and benefits in practice

33

Arguments for Arguments for evaluating evaluating organizational changeorganizational change Sound professional practice Sound professional practice Basis for organizational learning Basis for organizational learning Central to the development of Central to the development of

evidence based practiceevidence based practice Widespread cynicism about fads Widespread cynicism about fads

and fashionsand fashions To influence social or To influence social or

governmental policy governmental policy

44

Research and Research and evaluationevaluation Research focuses on relations between Research focuses on relations between

theory and empirical material (data)theory and empirical material (data)– Theory should provide a base for policy decisionsTheory should provide a base for policy decisions– Evidence can illuminate and inform theoryEvidence can illuminate and inform theory– Show what does not work as well as what doesShow what does not work as well as what does– Highlight areas of uncertainty and confusionHighlight areas of uncertainty and confusion– Demonstrate the complexity of cause-effect Demonstrate the complexity of cause-effect

relationsrelations– Understand predict controlUnderstand predict control

55

Pragmatic Evaluation: Pragmatic Evaluation: what matters is what what matters is what worksworks Why Why it works may be unclearit works may be unclear Knowledge increases complexityKnowledge increases complexity Reflexive monitoring of strategy links Reflexive monitoring of strategy links

to OL & KMto OL & KM Evidence and cultural contextEvidence and cultural context May be self fulfillingMay be self fulfilling Tendency to seek support for policyTendency to seek support for policy Extent of sound evidence unclearExtent of sound evidence unclear

66

Why is sound Why is sound evaluation so rare? evaluation so rare? Practice shows that evaluation is Practice shows that evaluation is

an extremely complex, difficult an extremely complex, difficult and highly political process in and highly political process in organizations.organizations.

Questions may be how many, not Questions may be how many, not what workswhat works

77

Evaluation modelsEvaluation models

1.1. Pre-evaluationPre-evaluation2.2. Goal based (Tyler, 1950)Goal based (Tyler, 1950)3.3. Realistic evaluation (Pawson & Tilley,1997; Realistic evaluation (Pawson & Tilley,1997;

Sanderson, 2002)Sanderson, 2002)4.4. ExperimentalExperimental5.5. Constructivist evaluation (Stake, 1975)Constructivist evaluation (Stake, 1975)6.6. Contingent evaluation (Legge, 1984)Contingent evaluation (Legge, 1984)7.7. Action learning (Reason & Bradbury, 2001) Action learning (Reason & Bradbury, 2001)

A study should be technically sound, A study should be technically sound, administratively convenient and politically administratively convenient and politically defensible. Alec Rodgerdefensible. Alec Rodger

88

1.1 Pre-evaluation 1.1 Pre-evaluation (Goodman & Dean, 1982)(Goodman & Dean, 1982)

The extent to which it is likely The extent to which it is likely that... A has an impact on bthat... A has an impact on b Scenario planningScenario planning Evidence based practiceEvidence based practice

– All current evidence thoroughly All current evidence thoroughly reviewed and synthesisedreviewed and synthesised

– Meta-analysisMeta-analysis– Systematic literature reviewSystematic literature review

Formative v summative Formative v summative (Scriven, 1967)(Scriven, 1967)

99

1.2 Pre-evaluation 1.2 Pre-evaluation issuesissues Based on theory and past Based on theory and past

evidence: not clear it will evidence: not clear it will generalise to the specific casegeneralise to the specific case

Formative: influences planningFormative: influences planning Argument: to understand a Argument: to understand a

system you must intervene system you must intervene (Lewin)(Lewin)

1010

2. 1. Goal based evaluation 2. 1. Goal based evaluation Tyler (1950)Tyler (1950)

Objectives used to aid planned changeObjectives used to aid planned change Can help clarify modelsCan help clarify models Goals from bench marking, theory or Goals from bench marking, theory or

pre-evaluation exercisespre-evaluation exercises Predict changesPredict changes Measure pre and post interventionMeasure pre and post intervention Identify the interventionsIdentify the interventions Were objectives achieved?Were objectives achieved?

1111

2.2 Difficulties with 2.2 Difficulties with Goal based evaluationGoal based evaluation

Who sets the goals? How do you identify Who sets the goals? How do you identify the intervention?the intervention?

– Tendency to managerialism (unitarist)Tendency to managerialism (unitarist)– Failure to accommodate value pluralismFailure to accommodate value pluralism– Over-commitment to scientific paradigmOver-commitment to scientific paradigm– What is measured gets doneWhat is measured gets done– No recognition of unanticipated effectsNo recognition of unanticipated effects– Focus on single outcome, not processFocus on single outcome, not process

1212

3.1 Realistic evaluation: 3.1 Realistic evaluation: Conceptual clarity Conceptual clarity (Pawson (Pawson & Tilley,1997)& Tilley,1997)

Evidence needs to be based on clear Evidence needs to be based on clear ideas about conceptsideas about concepts

Measures may be derived from theoryMeasures may be derived from theory Examine definitions used elsewhere Examine definitions used elsewhere Consider specific examples Consider specific examples Ensure all aspects are coveredEnsure all aspects are covered

1313

3.2 Realistic evaluation 3.2 Realistic evaluation Towards a theory: What Towards a theory: What are you looking for?are you looking for? Make assumptions and ideas explicit Make assumptions and ideas explicit

What is your theory of cause and effect?What is your theory of cause and effect?– What are you expecting to change What are you expecting to change

(outcome(outcome)? )? – How are you hoping to achieve this How are you hoping to achieve this

change (change (mechanismmechanism)?)?– What aspects of the What aspects of the contextcontext could be could be

important?important?

1414

3.3 Realistic evaluation 3.3 Realistic evaluation Context-mechanism-Context-mechanism-outcomeoutcome Context:Context: What environmental What environmental

aspects may affect the aspects may affect the outcome?outcome?– What else may influence the What else may influence the

outcomes?outcomes?– What other effects may there What other effects may there

be?be?

1515

3.4 Realistic evaluation 3.4 Realistic evaluation Context-mechanism-Context-mechanism-outcomeoutcome MechanismMechanism: What will you do to : What will you do to

bring about this outcome?bring about this outcome?– How will you intervene (if at all)?How will you intervene (if at all)?– What will you observe?What will you observe?– How would you expect groups to How would you expect groups to

differ?differ?– What mechanisms do you expect to What mechanisms do you expect to

operate?operate?

1616

3.5 Realistic evaluation 3.5 Realistic evaluation Context-mechanism-Context-mechanism-outcomeoutcome OutcomeOutcome: What effect or : What effect or

outcome do you aim for?outcome do you aim for?– What evidence could show it What evidence could show it

worked? worked? – How could you measure it? How could you measure it?

1717

4.1 Experimental 4.1 Experimental evaluation: evaluation:

Explain, predict and control by identifying causal Explain, predict and control by identifying causal relationshipsrelationships

Theory of causality makes predictions about Theory of causality makes predictions about variables eg training increases productivityvariables eg training increases productivity

Two randomly assigned matched groups: Two randomly assigned matched groups: experimental and controlexperimental and control

One group experiences intervention, one does One group experiences intervention, one does notnot

Measure outcome variable pre-test and post-Measure outcome variable pre-test and post-test (longitudinal)test (longitudinal)

Analyse for statistically significant differences Analyse for statistically significant differences between the two groupsbetween the two groups

Outcome linked back to modify theoryOutcome linked back to modify theory The gold standardThe gold standard

1818

4.2 Difficulties with 4.2 Difficulties with experimental evaluation experimental evaluation in organizationsin organizations Difficult to achieve in organizationsDifficult to achieve in organizations Unitarist viewUnitarist view Leaves out unforeseen effectsLeaves out unforeseen effects Problems with continuous change Problems with continuous change

processesprocesses Summative not formativeSummative not formative Generally at best quasi-Generally at best quasi-

experimentalexperimental

1919

5.1 Constructivist or 5.1 Constructivist or stakeholder evaluationstakeholder evaluation

Responsive evaluation Responsive evaluation (Stake, 1975)(Stake, 1975) or or Fourth generation evaluation Fourth generation evaluation (Guba & (Guba &

Lincoln, 1989)Lincoln, 1989) Constructivist interpretivist Constructivist interpretivist

hermeneutic methodologyhermeneutic methodology– Based on stakeholder claims concerns Based on stakeholder claims concerns

issuesissues– Stakeholders: agents, beneficiaries, Stakeholders: agents, beneficiaries,

victimsvictims

2020

5.2 Response to an IT 5.2 Response to an IT implementationimplementation(Brown, 1998)(Brown, 1998)

ThemeTheme The wardThe ward LaboratoLaboratoryry

IT TeamIT Team

GoalGoal Improve Improve quality to quality to patientspatients

Improve Improve quality for quality for ward staffward staff

Clinical Clinical and and financial financial benefitsbenefits

OutcomeOutcome Waste of Waste of time and time and energy on energy on a pointless a pointless systemsystem

No No improvemimprovement to ent to adequate adequate systemssystems

TechnicallTechnically y competent competent system - system - misconceimisconceived ved projectproject

2121

5.3 Constructivist 5.3 Constructivist evaluation issuesevaluation issues No one right answerNo one right answer Demonstrates complexity of Demonstrates complexity of

issues issues Highlights conflicts of interestsHighlights conflicts of interests Interesting for academics Interesting for academics Difficult for practitioners to Difficult for practitioners to

resolveresolve

2222

6 A Contingent approach to 6 A Contingent approach to evaluationevaluation(Legge, 1984)(Legge, 1984)

Do you want the proposed change Do you want the proposed change programme to be evaluated? programme to be evaluated? (Stakeholders)(Stakeholders)

What functions do you wish its What functions do you wish its evaluation to serve? evaluation to serve? (Stakeholders)(Stakeholders)

What are the alternative approaches to What are the alternative approaches to evaluation? evaluation? (Researcher)(Researcher)

Which of the alternatives best matches Which of the alternatives best matches the requirements?the requirements? (Discussion) (Discussion)

2323

7. Action research7. Action research

Identify good practiceIdentify good practice(Reason & Bradbury, 2001)(Reason & Bradbury, 2001) Action research Action research– Responds to practical issues in Responds to practical issues in

organizationsorganizations– Engages in collaborative relationshipsEngages in collaborative relationships– Draws on diverse evidenceDraws on diverse evidence– Value orientation - humanistValue orientation - humanist– Emergent, developmentalEmergent, developmental

2424

Problems with realist Problems with realist modelsmodels Tendency to managerialiseTendency to managerialise Over-commitment to scientific paradigmOver-commitment to scientific paradigm Context stripping, Context stripping, Over-dependence on measuresOver-dependence on measures Coerciveness: truth as non-negotiableCoerciveness: truth as non-negotiable Failure to accommodate value pluralismFailure to accommodate value pluralism Every act of evaluation is a political act, Every act of evaluation is a political act,

not tenable to claim it is value freenot tenable to claim it is value free

2525

Problems with Problems with Constructionist Constructionist approachapproach Evaluation judged by who for whom Evaluation judged by who for whom

and in whose interests?and in whose interests? Identify different views, then what?Identify different views, then what? Who has power?Who has power? Leaves decisions openLeaves decisions open May lead to ambiguityMay lead to ambiguity

2626

Why not evaluate?Why not evaluate?

Expensive in time and resourcesExpensive in time and resources De-motivating for individuals De-motivating for individuals Contradiction between “scientific” Contradiction between “scientific”

evaluation models and supportive, evaluation models and supportive, organization learning modelsorganization learning models

Individual identification with activityIndividual identification with activity Difficulties in objectifying and Difficulties in objectifying and

maintaining commitmentmaintaining commitment External evaluation ‘off the shelf’ External evaluation ‘off the shelf’

inappropriate and unhelpfulinappropriate and unhelpful

2727

Why evaluate?Why evaluate?(Legge, 1984)(Legge, 1984)

OvertOvert– Aids decision Aids decision

makingmaking– Reduce Reduce

uncertaintyuncertainty– Learn Learn – ControlControl

CovertCovert– Rally Rally

support/oppositionsupport/opposition– Postpone a Postpone a

decisiondecision– Evade Evade

responsibilityresponsibility– Fulfil grant Fulfil grant

requirementsrequirements– SurveillanceSurveillance

2828

ConclusionConclusion

Evaluation is very expensive, Evaluation is very expensive, demanding and complexdemanding and complex

Evaluation is a political process: need Evaluation is a political process: need for clarity about why you do itfor clarity about why you do it

Good evaluation always carries the risk Good evaluation always carries the risk of exposing failureof exposing failure

Therefore evaluation is an emotional Therefore evaluation is an emotional processprocess

Evaluation needs to be acceptable to Evaluation needs to be acceptable to the organizationthe organization

2929

Conclusion 2Conclusion 2

Plan and decide which model of Plan and decide which model of evaluation is appropriateevaluation is appropriate

Identify who will carry out the Identify who will carry out the evaluation and for what purposeevaluation and for what purpose

Do not overload the evaluation Do not overload the evaluation process:judgment or development?process:judgment or development?

Evaluation can give credibility and Evaluation can give credibility and enhance learningenhance learning

Informal evaluation will take place Informal evaluation will take place whether you plan it or notwhether you plan it or not