making a difference through program evaluation michael quinn patton september 21, 2010 program...

Making a DifferenceThrough Program Evaluation

Michael Quinn PattonSeptember 21, 2010

Program Evaluation Quality Assurance Summit

1

LBJ

2

3

Globalizationof the evaluationcommunity –of which you are all a part.

Burkina Faso

4

Evaluation

as a transdisciplineand a profession…

with quality assurance Standardsand professional developmentopportunities.

5

Standards – Quality AssuranceUtility – ensure relevance & useFeasibility – realistic, prudent,

diplomatic & frugalPropriety – ethical, legal, respectfulAccuracy – technically adequate to

determine merit or worth

For the full list of Standards:www.wmich.edu/evalctr/checklists/standardschecklist.htm

6

Meta-evaluation

Evaluating evaluationsfor

ongoing learning, improvement and quality assurance

7

Quality AssuranceEnsuring Value through

Evaluation

E-VALU-ate

8

Quality Assurance

Quality controland/or

Quality enhancement

9

Personal Factor:

Intended Useby

Intended Users

10

Critical success factors:

There are five key variables that are absolutely critical in evaluation use. They are, in order of importance:

• People–People

•People–People

»PEOPLE11

Big news

Evaluators are people too

12

Basic premise

Value and quality assurancein Evaluation

comes through EVALUATORS

13

Essential skills

Add value by being good at what you do

andknowing your roleand playing it well

14

Know thyself

Connais-toi toi-même

Greek: γνῶθι σεαυτόν

Latin: nosce te ipsum

15

Essential Competencies for Program Evaluators

Jean King, a recipient of the American Evaluation Association’s prestigious Alva and Gunnar MyrdalPractice Award, has worked for a number of years with colleagues and students conducting researchon and developing a framework for Essential Competencies for Program Evaluators (Ghere et al. 2006;King et al., 2001).

The final product is a taxonomy of essential program evaluator competencies organized into six primary categories.

16

Essential Competencies1. Professional Practice: knowing and observing

professional norms and values, including evaluation standards and principles.

2. Systematic Inquiry: expertise in the technical aspects of evaluations, such as design, measurement, data analysis, interpretation, and sharing results.

3. Situational Analysis: understanding and attending to the contextual and political issues of an evaluation, including determining evaluabiity, addressing conflicts, and attending to issues of evaluation use.

17

Essential Competencies4. Project Management: the nuts and bolts of

managing an evaluation from beginning to end, including negotiating contracts, budgeting, identifying and coordinating needed resources, and conducting the evaluation in a timely manner.

5. Reflective Practice: an awareness of one’s program evaluation expertise as well as the needs forprofessional growth.

18

Essential Competencies

6. Interpersonal Competence: the people skills needed to work with diverse groups of stakeholders to conduct program evaluations, including written and oral communication, negotiation, and cross-cultural skills.

19

Essential skills

Add value by being good at what you do

andknowing your roleand playing it well

20

The Nature of Expertise

and the role of expert

21

What’s it take to achieve world class expertise?

10,000 hours

22

Identities we take on

• Evaluator• Methodologist• Researcher• Auditor• Inspector• Learning facilitator• Judge

These roles are necessary, important and useful,but there’s another role that offers value:

The jester

23

24

SPEAKING TRUTH TO POWER

25

Jesting is Serious Businessbecause

Speaking TRUTH to Power is serious business

26

In the beginning…

27

28

About this book

Preview this book

Shake Hands with the Devil By Roméo Dallaire

This is a preview. The total pages displayed will

be limited. L

Loading...

Loading...

29

resultSearch Booksresultoe9S6SgfGeneral RPP1PP1ACfU3U1E

Value-added Challenge:Matching the evaluation process and design to the nature of the situation

to achieve

intended use by intended users:

Contingency-based

Evaluation 30

Evaluation’s General, Generic Value

Value of a Particular Evaluation

and Specific Approach31

Mrs. McCave and her 23 Daves

32

Not just experiencebut

Focused Practice and

REFLECTIONon experience

to achieve situational responsiveness 33

Story about ongoinglearning

34

35

Executing

Doing what we know how to do and know we should do

36

The Fine Art of Selling • Successful Selling Methods • What Makes a Great Salesperson?• The Secret to Selling• Increasing Sales Now• Sales Tips from Great Salesmen• Topping the Sales Charts• Breakthrough Strategies For Selling to Difficult People • Expert Hands-On Sales Techniques • Improving Sales Performance - Where to Begin?• Sales Techniques Revealed

37

Lessons from Sales Experts

1. Believe in your product2. Know your product3. Connect your product to what your

customer needs: listen

communicateconnect

38

Elevator speech test

Can you say clearly, in 30 seconds what your product is, why you believe in, and say it in a way that connects with the person you’re talking to?

39

Elevator test

Believe in your product

40

Bob Stake, “Beyond Neutrality: What Evaluators Care About”

41

Beyond Neutrality: What Evaluators Care About

1. We often care about the thing being evaluated.2. We, as evaluation professionals, care about evaluation.3. We advocate rationality.4. We care to be heard. We are troubled if our studies

are not used.5. We are distressed by underprivilege. We see gaps

among privileged patrons and managers andstaff and underprivileged participants and communities.

6. We are advocates of a democratic society.SOURCE: Robert Stake (2004:103–107).

42

Know your product

43

Value-added Challenge:Matching the evaluation

process and design to the nature of the situation to achieve intended use by

intended users

Contingency-based

Evaluation44

Connect your product to what your customer needs:

listencommunicate

connect

45

46

47

PROCESS USE

48

Three Cups of TeaBaltistani proverb:

First cup you share, you are a stranger.

Second cup, you are an honored guest.

Third cup, you are in relationship.

49

Enhancing Quality in Evaluation

1. Believe in evaluation2. Know evaluation (and yourself as

evaluator)3. Connect to your primary intended

users: listen

communicateconnect

50

51

52