Evaluating Evaluating Collaboration Collaboration

National Extension Family National Extension Family Life Specialists ConferenceLife Specialists Conference

April 28, 2005April 28, 2005

Ellen Taylor-Powell, Ph.D.Ellen Taylor-Powell, Ph.D.

Evaluation SpecialistEvaluation Specialist

University of Wisconsin-ExtensionUniversity of Wisconsin-Extension

Types of evaluation questionsTypes of evaluation questions

The outcomes are broad and complex. The outcomes are broad and complex. How do we get started? How do we get started?

Is evaluating process good enough or Is evaluating process good enough or do we have to evaluate outcomes?do we have to evaluate outcomes?

Who should be involved in evaluating Who should be involved in evaluating a collaborative program?a collaborative program?

I’m not in charge. How do I evaluate I’m not in charge. How do I evaluate it?it?

How do I take credit for something How do I take credit for something that we’ve done together?that we’ve done together?

Issues and challengesIssues and challenges

Power – controlPower – control Process of the evaluationProcess of the evaluation DataData

Standards and quality of the evaluationStandards and quality of the evaluation Cross-cultural issuesCross-cultural issues Measurements issuesMeasurements issues AttributionAttribution Taking creditTaking credit

Collaborative evaluationCollaborative evaluation(not evaluation of collaboration)(not evaluation of collaboration)

Since mid – 1970’s, new paradigm of pSince mid – 1970’s, new paradigm of participatory articipatory evaluation evaluation

“ “applied social research that involves trained applied social research that involves trained evaluation personnel…and practice-based decision evaluation personnel…and practice-based decision makers working in partnership” (makers working in partnership” (Cousins and Earl, 1992)Cousins and Earl, 1992)

Multiple approaches -from broadening decision Multiple approaches -from broadening decision making (practical) to emancipation and social making (practical) to emancipation and social change (transformation)change (transformation)

Emphasis on using data collection and feedback to Emphasis on using data collection and feedback to strengthen and monitor collaboration and thus strengthen and monitor collaboration and thus increase overall effectiveness and efficiencyincrease overall effectiveness and efficiency value in the process of evaluation, value in the process of evaluation, process useprocess use (Patton, (Patton,

1997), as much as product1997), as much as product

Who controls? Who participates? Who controls? Who participates? How much?How much?

Researcher control

Practitioner/participant control

Primary users

Deep participation All legitimate groups

Consultation

Adapted from Cousins and Whitmore, 1998

First…First…

Who wants to know what? Who wants to know what? For what purpose?For what purpose? How will information be used?How will information be used?

OUTCOMESINPUTS OUTPUTS

Collaborative Product

Collaborative Relationship

Assumptions External factors

Building a logic model of Building a logic model of collaborationcollaboration

EVALUATION

SITUATION

Collaborative

Effectiveness

Partners

Funding

Research-based

Key stake

holders

Change in behaviors

Value-added

• Clientele

•Users

•Policy makers

•Publics

Policy changes

Change in

Knowldge

Attitudes

Skills

Motivation

Intent

Self-efficacy

Implement activities – action plan

Monitor and evaluate

Communicate

Advocacy/Policy

Collaborative Relationship building

• Individual members

• Group

Change in• behaviors •decision making

Change in• KAS• Self-efficacy• Intent

•Effective functioning partnership•Member satisfaction

Changes in conditions

Collaboration: Theory of change Collaboration: Theory of change

WHAT DO YOU WANT TO KNOW?

System changes

Community changes

Capacity building -TA

Evaluating the Collaborative Evaluating the Collaborative Relationship Relationship

1. 1. Process evaluationProcess evaluation How is it functioning? How effective is the group How is it functioning? How effective is the group

work? Are we likely to achieve our desired work? Are we likely to achieve our desired results?results?

How satisfied are members? How satisfied are members? Questions about capacities, operations, climate, Questions about capacities, operations, climate,

contextcontext Factors influencing success Factors influencing success Projected tasks/activities relative to stages of Projected tasks/activities relative to stages of

developmentdevelopment Milestones and Critical Events (journey)Milestones and Critical Events (journey)

MILESTONESMILESTONES Significant points along the Significant points along the

wayway ExamplesExamples

Key stakeholders on boardKey stakeholders on board Vision statement Vision statement

establishedestablished Grant securedGrant secured Action plan formulated – Action plan formulated –

plan of workplan of work Project Project

implemented/service implemented/service providedprovided

Project evaluatedProject evaluated

CRITICAL EVENTSCRITICAL EVENTS Unexpected events, Unexpected events,

positive and negativepositive and negative Progress markersProgress markers Evidence of Evidence of

accomplishmentsaccomplishments Disruptions or obstaclesDisruptions or obstacles ExamplesExamples

Change in Change in membershipmembership

Policy changePolicy change New donor addedNew donor added

2. 2. Outcomes (Process outcomes):Outcomes (Process outcomes): What difference has being a part of this What difference has being a part of this

group made group made for the individualfor the individual? ? Knowledge, skills, motivations, behaviors, Knowledge, skills, motivations, behaviors,

etc. etc. Human capital developmentHuman capital development

What difference is their What difference is their for the groupfor the group? ? Group functioning, identify, resource pooling, Group functioning, identify, resource pooling,

etcetc

Note: Outcomes can be positive, negative or Note: Outcomes can be positive, negative or neutralneutral

MethodsMethods Informal feedbackInformal feedback Member (partner) SurveyMember (partner) Survey Member (partner) interviewsMember (partner) interviews Group discussionsGroup discussions Key informant interviewsKey informant interviews ObservationObservation Identification and use of indicatorsIdentification and use of indicators Network analysis ; sociogramNetwork analysis ; sociogram Use existing materials (integrate into ongoing Use existing materials (integrate into ongoing

operations)operations) Minutes of meetingsMinutes of meetings Logs: telephone, event, registration formsLogs: telephone, event, registration forms Management chartsManagement charts

WHEN?

Periodic Review

Points of particular concern

Tools - TechniquesTools - Techniques

Community Group Member SurveyCommunity Group Member Survey Collaborative Relationship scalesCollaborative Relationship scales Internal collaborative functioning scalesInternal collaborative functioning scales Plan Quality IndexPlan Quality Index Meeting effectiveness inventoriesMeeting effectiveness inventories Stage of readinessStage of readiness On-line Wilder Collaboration Factors On-line Wilder Collaboration Factors

Inventory (Amherst H. Wilder Foundation)Inventory (Amherst H. Wilder Foundation) On-line Partnership self-assessment tool On-line Partnership self-assessment tool

(Center for Advancement of Collaborative (Center for Advancement of Collaborative Strategies in Health)Strategies in Health)

Evaluating Programs/Products Evaluating Programs/Products created/implemented by the created/implemented by the

collaborationcollaboration1.1. Process or implementation Process or implementation

evaluationevaluation

(Focus: program delivery vs. (Focus: program delivery vs. coordination or support role)coordination or support role)

How is program being How is program being implemented? Fidelity to plan? implemented? Fidelity to plan? Extent of delivery? Participation? Extent of delivery? Participation? What is/has happened that wasn’t What is/has happened that wasn’t planned?planned?

Outcome evaluationOutcome evaluation What is different? For whom? How? What is different? For whom? How?

To what extent?To what extent? For: Individuals, Groups/Families, For: Individuals, Groups/Families,

Agencies, Systems, CommunitiesAgencies, Systems, Communities Changes in …Changes in …

IndividualsIndividuals Attitudes, perceptions, knowledge, Attitudes, perceptions, knowledge, competence, skills, abilities, competence, skills, abilities, behaviors, actions, lifestylesbehaviors, actions, lifestyles

Groups/Groups/familiesfamilies

Interactions, behaviors, actions, Interactions, behaviors, actions, values, culturevalues, culture

Agency, Agency, organizationorganization

#/type of services/programs #/type of services/programs delivered, access, practices, resource delivered, access, practices, resource generation, resource use, policiesgeneration, resource use, policies

SystemsSystems Relationships, interaction patterns, Relationships, interaction patterns, linkages, networks, practices, linkages, networks, practices, policies, resource use, policies, resource use, institutionalization of changesinstitutionalization of changes

CommunitieCommunitiess

Values, attitudes, relations, support Values, attitudes, relations, support systems, civic action, social norms, systems, civic action, social norms, policies, laws, practices, conditionspolicies, laws, practices, conditions

Change in :

Tools - TechniquesTools - Techniques

Monitor implementationMonitor implementation Logs, management charts,Logs, management charts, InterviewsInterviews ObservationsObservations

Achievement of outcomesAchievement of outcomes Clientele surveysClientele surveys Clientele interviewsClientele interviews ObservationsObservations

Mixed MethodsMixed Methods

Evaluating self - Taking creditEvaluating self - Taking credit

Mutual (reciprocal) accountabilityMutual (reciprocal) accountability How do I take credit for my part? How How do I take credit for my part? How

does Extension gain visibility, does Extension gain visibility, recognition?recognition?

What is your contribution? What role did What is your contribution? What role did you play? What value did you bring? you play? What value did you bring?

Document role you play, your activities Document role you play, your activities and contributions, inputs you bring, and contributions, inputs you bring, resources you make available, niche, resources you make available, niche, value…value…

Your contributionYour contribution Log of activities, roles playedLog of activities, roles played Record inputs, resources contributedRecord inputs, resources contributed Management chart; analysis of Management chart; analysis of

minutesminutes Independent assessmentIndependent assessment

SurveySurvey InterviewsInterviews

Your (partner) performance: Your (partner) performance: Most important indicator: other Most important indicator: other

partners’ satisfaction with your partners’ satisfaction with your performance (Brinkerhoff, 2002)performance (Brinkerhoff, 2002)

Mutual assessment among partners Mutual assessment among partners of each partner’s performance. Resulting of each partner’s performance. Resulting discussion re. Discrepancies = powerful discussion re. Discrepancies = powerful information sharing and trust building. information sharing and trust building.

(We aren’t very good at this type of thing)(We aren’t very good at this type of thing)

Web addressWeb address

http://www.uwex.edu/ces/pdandehttp://www.uwex.edu/ces/pdande