impact evaluation: initiatives, activities, & coalitions stephen horan, phd community health...

Impact Evaluation: Impact Evaluation: Initiatives, Activities, & Initiatives, Activities, &

CoalitionsCoalitions

Stephen Horan, PhDStephen Horan, PhD

Community Health Solutions, Community Health Solutions, Inc.Inc.

September 12, 2004September 12, 2004

22

ObjectivesObjectives

Participants will learn:Participants will learn:– 1. When an evaluation should be 1. When an evaluation should be

conductedconducted– 2. What questions to ask to address 2. What questions to ask to address

outcomes & impactoutcomes & impact– 3. Strategies for evaluating outcomes & 3. Strategies for evaluating outcomes &

impactimpact– 4. Strategies for incorporating multiple 4. Strategies for incorporating multiple

programs and activitiesprograms and activities– 5. Strategies for incorporating community 5. Strategies for incorporating community

input input

33

Ground RulesGround Rules

Evaluation talk can be confusing – don’t let Evaluation talk can be confusing – don’t let semantics get in the waysemantics get in the way– This presentation defines This presentation defines

Outcomes as changes in health program clientsOutcomes as changes in health program clients Impacts as changes in the organization, system, or Impacts as changes in the organization, system, or

communitycommunity Different funders or textbooks might define these Different funders or textbooks might define these

terms differentlyterms differently

– This presentation uses a model of personal This presentation uses a model of personal change based on knowledge, attitudes & change based on knowledge, attitudes & beliefs, skills, behaviors, and health status.beliefs, skills, behaviors, and health status. There are other equally valid models – no one is There are other equally valid models – no one is

saying you have to use this one!saying you have to use this one!

44

When to evaluateWhen to evaluate

1. When a funder requires it1. When a funder requires it 2. When the program is experimental2. When the program is experimental 3. When the program is controversial3. When the program is controversial 4. When the program is expensive4. When the program is expensive 5. When you think the program might 5. When you think the program might

not be worth itnot be worth it 6. When you think the program is worth 6. When you think the program is worth

it but might be in danger of losing it but might be in danger of losing fundingfunding

55

Questions for addressing Questions for addressing outcomes & impactoutcomes & impact

66


Five phases of evaluationFive phases of evaluation– 1. Context evaluation1. Context evaluation– 2. Program design evaluation2. Program design evaluation– 3. Program implementation or process 3. Program implementation or process

evaluationevaluation– 4. Program outcome evaluation4. Program outcome evaluation– 5. Program impact evaluation5. Program impact evaluation

77


1. Context evaluation1. Context evaluation– Why is this program needed?Why is this program needed?– Does this program duplicate other Does this program duplicate other

programs?programs?– Does this program conflict with other Does this program conflict with other

programs?programs?– How might public policy affect this How might public policy affect this

program?program?– Does this program fit our funder’s criteria?Does this program fit our funder’s criteria?

88


2. Program design evaluation2. Program design evaluation– What is our program theory?What is our program theory?– Who is our target population?Who is our target population?– What are the needs of our target population?What are the needs of our target population?– What outcomes will we help out target What outcomes will we help out target

population achieve?population achieve?– What services will we provide?What services will we provide?– How many people will we serve?How many people will we serve?– What resources will we need to run the What resources will we need to run the

program?program?

99


3. Program implementation evaluation3. Program implementation evaluation– How many people are we serving?How many people are we serving?– What services are they receiving?What services are they receiving?– Are they receiving the services they Are they receiving the services they

should?should?– What activities are we conducting?What activities are we conducting?– How much are we spending?How much are we spending?– All of the above compared to Program All of the above compared to Program

DesignDesign

1010

Questions for addressingQuestions for addressingoutcomes & impactoutcomes & impact

4. Program 4. Program outcomeoutcome evaluation evaluation– How have our How have our clientsclients changed in terms changed in terms

of:of: KnowledgeKnowledge Attitudes & beliefsAttitudes & beliefs SkillsSkills BehaviorsBehaviors Health statusHealth status

1111


5. Program 5. Program impactimpact evaluation evaluation– How has our How has our organizationorganization or or communitycommunity

changed in terms of:changed in terms of: FinancingFinancing Service capacityService capacity Organization of servicesOrganization of services Quality of servicesQuality of services Accessibility of servicesAccessibility of services Cost of servicesCost of services Community health statusCommunity health status

1212

Strategies for evaluating Strategies for evaluating outcomes & impactoutcomes & impact

1313


1. Begin with your program theory1. Begin with your program theory 2. Develop your program design2. Develop your program design 3. Capture your program design in a 3. Capture your program design in a

logic modellogic model 4. Decide which outcomes & impacts 4. Decide which outcomes & impacts

to measureto measure 5. Negotiate outcomes & impacts 5. Negotiate outcomes & impacts

with your funderwith your funder

1414


1. Begin with your program theory1. Begin with your program theory– Program theory is your set of Program theory is your set of

assumptions about cause and effect assumptions about cause and effect between our program and are target between our program and are target population’s outcomespopulation’s outcomes

– Most often overlooked component of Most often overlooked component of program evaluationprogram evaluation

– What makes you think this will work?What makes you think this will work?

1515


2. Capture your program design in a 2. Capture your program design in a program logic modelprogram logic model– Target populationTarget population– Assumptions (program theory)Assumptions (program theory)– InputsInputs– ActivitiesActivities– OutputsOutputs– OutcomesOutcomes– ImpactImpact

1616


3. Develop your program design3. Develop your program design– Specify your target populationSpecify your target population– Specify your program theory in a set of Specify your program theory in a set of

assumptionsassumptions– Specify your target impactSpecify your target impact– Specify your target outcomesSpecify your target outcomes– Specify your target outputsSpecify your target outputs– Specify your required activitiesSpecify your required activities– Specify your required resourcesSpecify your required resources

1717


4. Decide which outcomes & impacts 4. Decide which outcomes & impacts to measureto measure– Is this outcome or impact important to our Is this outcome or impact important to our

audiences?audiences?– Is this outcome or impact within our Is this outcome or impact within our

sphere of influence?sphere of influence?– Is this outcome or impact achievable Is this outcome or impact achievable

within the timeframe of the reporting within the timeframe of the reporting period?period?

– Is this outcome or impact measurable?Is this outcome or impact measurable?

1818


5. Negotiate with your funder 5. Negotiate with your funder – Not every outcome or impact can be measuredNot every outcome or impact can be measured– Not every outcome or impact should be measuredNot every outcome or impact should be measured– Arm yourself with knowledgeArm yourself with knowledge

Your program theoryYour program theory Why a particular outcome or impact should or should not Why a particular outcome or impact should or should not

be measuredbe measured What it will cost to measure a particular outcome or What it will cost to measure a particular outcome or

impactimpact

– If they insist, tell them what you think the If they insist, tell them what you think the evaluation will cost, and ask them to either fund it evaluation will cost, and ask them to either fund it or suggest a cheaper way to do the evaluationor suggest a cheaper way to do the evaluation

1919

Strategies for incorporating Strategies for incorporating multiple programs and multiple programs and

activitiesactivities

2020

Strategies for incorporating Strategies for incorporating multiple programs and multiple programs and

activitiesactivities 1. Get everyone to agree on:1. Get everyone to agree on:

– Program theoryProgram theory– Target outcomesTarget outcomes– Target impactsTarget impacts

2. Get everyone to develop a program design 2. Get everyone to develop a program design complete with logic modelcomplete with logic model

3. Agree upon common outcome indicators3. Agree upon common outcome indicators 4. Agree upon common impact indicators4. Agree upon common impact indicators 5. Share data collection tools5. Share data collection tools 6. Share data analysis methods6. Share data analysis methods 7. Pool the data or agree on a standard format for 7. Pool the data or agree on a standard format for

aggregating the dataaggregating the data

2121

Strategies for incorporating Strategies for incorporating community inputcommunity input

2222

Strategies for incorporating Strategies for incorporating community inputcommunity input

1. Ask community members to review and 1. Ask community members to review and comment on your program logic modelcomment on your program logic model

2. Ask community members to review and 2. Ask community members to review and comment on your evaluation workplancomment on your evaluation workplan

3. Ask community members to pilot test 3. Ask community members to pilot test your data collection instrumentsyour data collection instruments

4. Survey your community on its 4. Survey your community on its perceptions of your programperceptions of your program

5. Ask community members to help you 5. Ask community members to help you interpret your evaluation resultsinterpret your evaluation results

2323

Additional resourcesAdditional resources

2424

Additional resourcesAdditional resources

For additional ideas, guidelines, and For additional ideas, guidelines, and connections, go to connections, go to www.CommunityHealthResource.netwww.CommunityHealthResource.net

In particular:In particular:– Go to the Program Design page and Go to the Program Design page and

Program Evaluation page and download Program Evaluation page and download worksheetsworksheets

2525

PresenterPresenter

Stephen Horan, PhDStephen Horan, PhD PresidentPresident Community Health Solutions, Inc.Community Health Solutions, Inc. 9603 BC Gayton Road Suite 2019603 BC Gayton Road Suite 201 Richmond, VA 23238Richmond, VA 23238 T: 804.673.0166T: 804.673.0166 F: 804.673.2244F: 804.673.2244 [email protected]@communityhealthinfo.com

impact evaluation: initiatives, activities, & coalitions stephen horan, phd community health...

Documents

program design evaluation

program outcome evaluation

program theory

program conflict

program design slide

context evaluation

phases of evaluation

outcomes impact slide