20090601 quinn patton michael a

7/28/2019 20090601 Quinn Patton Michael A

1/57

Michael Quinn PattonCES, 2009

Dev elo p m en tal Ev alu at ion with

Michael Quinn Patton

CESJune 1, 2009

1


2/57


Evolving Understandings

I keep changing what I said. Any person

who is intellectually alive changes hisideas. If anyone at a university isteaching the same thing they wereteaching five years ago, either the fieldis dead, or they havent been thinking.

Noam Chomsky

The Professor Provaocateur, The New York Times Magazine, Nov. 2, 2003: 13.2


3/57


Interpretive Frameworks

May 2003 Harvard Business Review "The

High Cost of Accuracy." Kathleen Sutcliffeand Klaus Weber.

They concluded that " the way senior executives interpret their business

environment is more important for performance than how accurately theyknow their environment ."

3


4/57


They further concluded that it is a waste of

resources to spend a lot of moneyincreasing the marginal accuracy of dataavailable to senior executives compared tothe value of enhancing their capacity tointerpret whatever data they have .

Executives were more limited by a lack of capacity to make sense of data than byinadequate or inaccurate data.In essence, they found that interpretivecapacity, or "mind-sets," distinguish high-

performance more than data quality andaccuracy. 4


5/57


Original Primary Options

Formativeand

SummativeEvaluation

(Mid-term and End-of-Project Reviews)

5


6/57


ImprovementversusDevelopment

6


7/57


Evidence-based Practice

Evaluation grew up in the projects

testing models under a theory of change that pilot testing would lead toproven models that could bedisseminated and taken to scale :

The search for b es t p rac t ices and ev id en c ed -b as ed p rac t ices

7


8/57


Fundamental Issue:How the World Is Changed

Top-down dissemination of

proven modelsversus

Bottoms-up adaptivemanagement

8


9/57


Models vs. Principles

Identifying proven principles for adaptive management

(bottoms-up approach)versus

Identifying and disseminatingproven models

(top down approach)9


10/57


Conditions that challenge traditional

model-testing evaluation

High innovation Development

High uncertainty Dynamic

Emergent Systems Change

AdaptiveManagement

10


11/57


Mintzberg on StrategyTwo types of strategy: Intended & Emergent

Unrealized

StrategyIntended

Strategy DeliberateStrategyRealized

Emergent StrategyStrategy

11


12/57


Re-conceptualizing Use

Use is a process not a event Use involves an interaction not just

a report Use involves training for use not just delivery of results

Use is a leadership function

12


13/57


Some Evaluation Premises : Evaluation is part of initial program design, including

conceptualizing the theory of change Evaluators role is to help users clarify their purpose,

hoped-for results, and change model. Evaluators can/should offer conceptual and

methodological options. Evaluators can help by questioning assumptions. Evaluators can play a key role in facilitating

evaluative thinking all along the way. Interpretative dialogue is critical. Designs can be emergent and flexible.

13


14/57


Contingency-based

Evaluation Situational analysis & responsiveness

Context sensitivity Clarify and focus on intended users:

stakeholder analysis Clarify and focus on intended uses Methodological appropriateness Criteria for evaluating the evaluation:

credibility, meaningfulness

14


15/57


Seeing Through AComplexity Lens

You don't see something until you have the right metaphor to let you perceive it. Thomas Kuhn 15


16/57


Get t in g to May b e:

How th e World Is Changed? 2006

Frances Westley, BrendaZimmerman, Michael Q. Patton

Random House Canada ,

16


17/57


Conceptual Options

Simple

Complicated

Complex

17


18/57


Types of Community IssuesThe Certainty/Agreement Matrix

CertaintyClose to Far from C l o s e

t o

18


19/57


20/57



Simple (Known arena of action) Tight, centralized connections. Can identify and make sense of patterns. Linear cause and effect.

Best practices identifiable within the currentcontext (which of course may not beself-evident or known to others henceimportance of context).

20


21/57


Following a Recipe A Rocket to the Moon Raising a Child

Complicated Complex

] The recipe is essential

]Recipes are tested toassure replicability of

later efforts

] No particular

expertise; knowing howto cook increasessuccess

] Recipes producestandard products

] Certainty of sameresults every time

Simple

21


22/57



Complicated (Knowable arena)Relationships are looser but still

clustered around a central core.Cause and effect is dynamic, multi-

dimensional, and enmeshed in

system relationships. System relationships can be

modelled and understood.Expertise and coordination needed.

22


23/57


Following a Recipe A Rocket to the Moon Raising a

Child Formulae arecritical andnecessary

Sending one rocketincreasesassurance that nextwill be ok

High level of expertise in manyspecialized fields +coordination

Rockets similar incritical ways

High degree of certainty of outcome

Complicated Complex


] Recipes are tested toassure replicability of later efforts

] No particularexpertise; knowing howto cook increasessuccess



Simple

23


24/57


Socially complicated

Implementing human rights

agreements, like gender equity or outlawing child labor

Environmental InitiativesMany different and competingstakeholdersDiverse vested interestsHigh stakes

24


25/57


Socially complicatedsituations

pose the challengeof coordinating and

integrating

many players

25


26/57


Stakeholder MappingHigh Interest/

Low Power

THE INVOLVED

High Interest/High Power

THE PLAYERS

THE CROWD

Low interest/Low Power

CONTEXT SETTERS

Low Interest/High Power

26


27/57



ComplexCentre is loosely connected to network.Cause effect difficult to track; nonlinear,

interdependent relationshipsHighly context dependent.Outcomes emergent, not predictable.

27


28/57


Following a Recipe A Rocket to the Moon Raising a

Child

Sending one rocketincreasesassurance that nextwill be ok

High level of expertisein many specializedfields +coordination


High degree of

certainty of outcome

Formulae have onlya limited application

Raising one childgives no assuranceof success with thenext

Expertise can helpbut is not sufficient;relationships arekey

Every child isunique

Uncertainty of outcome remains

Complicated Complex






Simple

28


29/57


Complex Nonlinear Dynamics Nonlinear: Small actions can have large

reactions. The Butterfly Wings Metaphor Emergent: Self-organizing, Attractors Dynamic: Interactions within, between,

and among subsystems and parts withinsystems can volatile, changing

Getting to Maybe: Uncertainty,unpredictable, uncontrollable

29

C li d C l


30/57


Following a Recipe A Rocket to the Moon Raising a Child Formulae are critical

and necessary

Sending one rocketincreases assurancethat next will be ok

High level of expertise in manyspecialized fields +coordination

Separate into partsand then coordinate


High degree of certainty of outcome

Formulae have only alimited application

Raising one childgives no assurance of success with the next

Expertise can helpbut is not sufficient;relationships arekey

Cant separate partsfrom the whole

Every child is unique Uncertainty of

outcome remains

Complicated Complex




] Recipe notes thequantity and nature of

parts needed



Simple

30


31/57


A Leader's Framework for DecisionMaking by David J. Snowden andMary E. Boone, Harvard B us iness Review,

November, 2007:Wise executives tailor their approach to fit

the complexity of the circumstancesthey face.

31


32/57


Example

The McGill-McConnell Leadership

Program ExampleSimple elements

Complicated elements

Complex elements

32


33/57

Simple outcomes

Increase knowledge and skills of

participants

Evaluation: Pre-post data anddocumentation of learning


33


34/57

Complicated Impacts

Change participants organizations

Evaluation:

Case studiesof

organizational change


34


35/57

Complex Vision

Infuse energy into the moribund not-

for-profit (voluntary) sector Make the sector more dynamic

Create network of leaders whoactively engage in change


35


36/57

Evaluating the Complex

Real time follow-up of network

connections and actions Follow-up is an intervention

Rapid feedback of findings permitsinfusion of resources in support of emergent outcomes


36


37/57


Process Use

Infusing ev aluat ive th in k ing as a

primary type of process use.

Capacity-building as anevaluation focus of

process use .

37


38/57

Paradigms and Lenses

The importance of interpretive

frameworks Complexity as an interpretive

framework


38


39/57


Complex Situations Highly emergent (difficult to

plan and predict) Highly dynamic, rapidly

changing Relationships are

interdependent and non-linear rather than simple and linear (cause-effect) 39


40/57


Contingency-basedDevelopmental

Evaluation

40


41/57

DEVELOPMENTAL EVALUATION

DEFINEDEvaluation processes, including asking evaluative

questions and applying evaluation logic, to support program, product, staff and/or organizationaldevelopment . The evaluator is part of a team whose

members collaborate to conceptualize, design and testnew approaches in a long-term, on-going process of continuous improvement, adaptation and intentional

change . The evaluator's primary function in the teamis to elucidate team discussions with evaluativequestions, data and logic, and facilitate data-based

decision-making in the developmental process.Michael Quinn Patton

CES, 200941


42/57


Other names

Real time evaluation Emergent evaluation Action evaluation Adaptive evaluation

42


43/57


CONTRASTS

Traditional

evaluations Testing models

Complexity-based,

DevelopmentalEvaluation

Supportinginnovation andadaptation

43


44/57


TraditionalEvaluation

Render definitive

judgments of success or failure


Provide feedback,generate learnings,support directionor affirm changesin direction in real

time

44


45/57


TraditionalEvaluation

Render definitive judgments of success or failure

Measure success

againstpredetermined goals


Provide feedback, generatelearnings, support direction or affirm changes in direction

Develop newmeasures andmonitoringmechanisms asgoals emerge &evolve

45


46/57


Traditional Developmental

Evaluation Evaluation Evaluator external,

independent,objective

Evaluator part of a

team, a facilitator and learning coachbringing evaluative

thinking to the table,supportive of theorganizations goals

46


47/57



Evaluation Evaluation Evaluator

determines thedesign based on theevaluators

perspective aboutwhat is important.The evaluator

controls theevaluation.

Evaluator

collaborates withthose engaged inthe change effort to

design anevaluation processthat matches

philosophically andorganizationally.

47


48/57



Evaluation Evaluation Design the

evaluation based onlinear cause-effectlogic models

Design the

evaluation tocapture systemdynamics,

interdependencies,and emergentinterconnections

48


49/57



Evaluation Evaluation Aim to produce

generalizablefindings across time& space .

Aim to produce

context-specificunderstandings thatinform ongoing

innovation

49


50/57



Evaluation Evaluation Accountability

focused on anddirected to externalauthorities and

funders.

Accountability

centered on theinnovators deepsense of

fundamental valuesand commitments and learning.

50


51/57



Evaluation Evaluation Accountability to

control and locateblame for failures

Learning to respond

to lack of controland stay in touchwith whats

unfolding And thereby

responds t ra tegical ly

51


52/57


53/57



Evaluation Evaluation Evaluation

engendersfear o f fa i lu re .

Evaluation supports

hu ng er for l earn ing .

53


54/57


Conditions

High innovation Development

High uncertainty Dynamic

Emergent

54


55/57


Challenge:Matching the evaluationprocess and design to thenature of the situation:

Contingency-based

Evaluation

55


56/57


References

Getting to Maybe: How the World Is Changed?

Frances Westley, Brenda Zimmerman, MichaelQ. Patton, Random House Canada, 2006

Utilization-Focused Evaluation , 4 th ed.,Michael Quinn Patton, Sage, 2008.

Developmental Evaluation , Michael Quinn Patton,Guilford Press, forthcoming.

56


57/57

Resources

The J.W. McConnell Foundation onDevelopmental Evaluation

http://www.mcconnellfoundation.ca/default.aspx?page=139

h l

20090601 quinn patton michael a

Documents