fundamentals of evaluation for public health programs of evaluation for public health...
TRANSCRIPT
![Page 1: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/1.jpg)
Fundamentals of Fundamentals of Fundamentals of Fundamentals of Evaluation for Public Evaluation for Public Evaluation for Public Evaluation for Public Health ProgramsHealth ProgramsHealth ProgramsHealth Programs
1
![Page 2: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/2.jpg)
2
![Page 3: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/3.jpg)
ObjectivesObjectivesObjectivesObjectives
By the end of the workshop, participants will be able to:
• Define common evaluation terms
• Explain the difference between outcome and process evaluation
• Describe methods for different evaluation
• Construct evaluation indicators
• Identify resources to assist with evaluation
3
![Page 4: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/4.jpg)
Definition of EvaluationDefinition of EvaluationDefinition of EvaluationDefinition of Evaluation
The systematic application of methods to collect and analyze information about the activities,
characteristics, and outcomes of programs with the intent of furthering its development and
improvement
4
![Page 5: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/5.jpg)
Reasons Reasons Reasons Reasons to Evaluateto Evaluateto Evaluateto Evaluate
• Accountability to the funder, to the staff, to the clients, and
to the community
• Evaluation can tell us if the most vulnerable populations are
receiving appropriate and effective services
• Demonstrate effectiveness to funders, administration,
community stakeholders, community leadership, clients
• Improve implementation and effectiveness of programs
• Better manage limited resources
• Document program accomplishments
• Justify current program funding
• Document program development and activities to help
ensure successful replication and future planning
5
![Page 6: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/6.jpg)
It It It It IIIIs s s s AAAAll ll ll ll CCCConnectedonnectedonnectedonnected
6
Program Implementation
Program Evaluation
Program Development
![Page 7: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/7.jpg)
Relationship Between Planning, Relationship Between Planning, Relationship Between Planning, Relationship Between Planning, Implementation, and OutcomesImplementation, and OutcomesImplementation, and OutcomesImplementation, and Outcomes
7
Planning Implementation Outcomes
![Page 8: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/8.jpg)
Implications of Not Knowing How Implications of Not Knowing How Implications of Not Knowing How Implications of Not Knowing How an Intervention Was Implementedan Intervention Was Implementedan Intervention Was Implementedan Intervention Was Implemented
8
Planning ??????? Outcomes
![Page 9: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/9.jpg)
9
Formative Evaluation◦ Collects data describing
the needs of the population and the factors that put them at risk
Answers questions such as:
• How should the intervention be designed
or modified to address population needs?
• Is this standardized program appropriate
for the needs of our community?
![Page 10: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/10.jpg)
10
Process Monitoring◦ Collects data describing
the characteristics of the population served, the services provided, and the resources used to deliver those services
Answers questions such as:
• What services were delivered?
• Who was served?
• What resources were used?
![Page 11: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/11.jpg)
11
Process Evaluation◦ Collects and analyzes
detailed data about how the program was delivered, differences between the intended population and the population served, and access to the intervention
Answers questions such as:
• Was the intervention implemented as intended?
• Did the program reach the intended audience?
• What barriers did clients experience in accessing
the program?
• How did the population perceive the quality of
the program?
![Page 12: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/12.jpg)
12
Outcome Monitoring◦ Collects information about
client outcomes before and after the intervention, such as knowledge, attitudes, skills, or behaviors
Answers questions such as:
• What changes took place?
• What changes did our participants
experience?
• Did the expected outcomes occur?
![Page 13: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/13.jpg)
13
Outcome Evaluation◦ Collects data about
outcomes before and after the intervention for clients as well as with a similar group that did not participate in the intervention being evaluated
Answers questions such as:
• Did the program cause the outcomes?
• How replicable is this?
![Page 14: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/14.jpg)
14
Impact Evaluation◦ Collects data about health
conditions at the reservation, state, regional, and/or national levels
Answers questions such as:
• What long-term effects do all of the
related programming have on the health
condition?
![Page 15: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/15.jpg)
Evaluation Related to Planning, Evaluation Related to Planning, Evaluation Related to Planning, Evaluation Related to Planning, Implementation, and OutcomesImplementation, and OutcomesImplementation, and OutcomesImplementation, and Outcomes
15
Planning Implementation Outcomes
Formative Evaluation
Process Monitoring
Process Evaluation
Outcome Monitoring
Outcome Evaluation
Impact Evaluation
![Page 16: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/16.jpg)
Building Upon One AnotherBuilding Upon One AnotherBuilding Upon One AnotherBuilding Upon One Another
16
Planning Effective Programs and Interventions
Determining what Services
were Delivered to Whom
Determining if Program
was Implemented as Intended
Determining if Program
Caused Outcomes
Determining if Program
Achieved its Outcome Objectives
Determining
Broader ImpactsImpact Evaluation
Outcome Evaluation
Outcome Monitoring
Process Evaluation
Process Monitoring
Formative Evaluation
![Page 17: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/17.jpg)
Data CollectionData CollectionData CollectionData Collection
Process Monitoring
Sign-in Sheets
Demographic & Participant Information
Participant Counts
Process Evaluation
Satisfaction Surveys &
Questionnaires
Focus Group
Comparison of what happened
vs intent
Outcome Monitoring
Pre-post Tests on Knowledge, Attitudes, etc.
Case Reviews
Long term Follow-up
Outcome Evaluation
Experimental and Control
Groups
Replication
Using Same Outcome
Monitoring17
![Page 18: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/18.jpg)
• Measures◦ How and what do we measure so that we can show if we
were successful?◦ The number of people enrolling in treatment
◦ The knowledge of community members
• Indicators◦ What indicates that we were successful in our program?
◦ An increase in the number of people enrolling in methamphetamine treatment.
◦ An increase in the knowledge of community members regarding the impact of methamphetamine on the body
18
![Page 19: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/19.jpg)
Kinds of IndicatorsKinds of IndicatorsKinds of IndicatorsKinds of Indicators
Process Indicators
• Output Indicators
Outcome Indicators
• Short Term
• Long Term
19
![Page 20: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/20.jpg)
CASE STUDYCASE STUDYCASE STUDYCASE STUDY
• We are designing a project that will target methamphetamine prevention through the implementation of a structured 8-session intervention. We will recruit 10-12 youth to attend the intervention sessions and go through all 8 sessions. The intervention is designed to teach them about methamphetamine and raise their confidence to say no to peer pressure to use. We will have a series of 3 total cohorts (for a total of 30-36 youth).
20
![Page 21: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/21.jpg)
Evaluation PlanningEvaluation PlanningEvaluation PlanningEvaluation Planning
• Example:◦ Process Indicator: An increase in the number of people
enrolling in the intervention◦ Process Objective: By the end of the first year, there will be a
10% increase in the number of youth enrolling in the intervention
• Example:◦ Outcome Indicator: An decrease in the self-reported use of
methamphetamine by youth on the Reservation◦ Outcome Objective: Within two years after the end of the
intervention, there will be a 20% decrease in the self-reported use of methamphetamine by youth between the ages of 13 and 18 on X Reservation
21
![Page 22: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/22.jpg)
Activity: Activity: Activity: Activity: Let’s Evaluate this Workshop!Let’s Evaluate this Workshop!Let’s Evaluate this Workshop!Let’s Evaluate this Workshop!
22
Outcome Evaluation
Outcome Monitoring
Process Evaluation
Process Monitoring
Formative Evaluation
Activities or tools
Questions to ask
![Page 23: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/23.jpg)
23
Evaluation PlanningEvaluation PlanningEvaluation PlanningEvaluation Planning
![Page 24: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/24.jpg)
Strategies to Promote UtilizationStrategies to Promote UtilizationStrategies to Promote UtilizationStrategies to Promote Utilization
• Develop buy-in among evaluation stakeholders
• Clearly identify the intended users of the evaluation data
• Identify evaluation questions meaningful to the intended users
• Decide how the data will be used before the evaluation is conducted
• Present data in a user-friendly format
• Define clear responsibilities for data collection, analysis, storage and reporting
24
![Page 25: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/25.jpg)
Evaluation Technical Evaluation Technical Evaluation Technical Evaluation Technical Assistance ResourcesAssistance ResourcesAssistance ResourcesAssistance Resources
• National Indian Health Board
• NCUIH
• Tribal Epidemiology Centers
• Websites
• Volunteers (e.g., skilled board members)
• TCU and university faculty
• Program officers
• Evaluation consultants
25
![Page 26: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/26.jpg)
Working with an Evaluation Working with an Evaluation Working with an Evaluation Working with an Evaluation ConsultantConsultantConsultantConsultant
• Select a consultant who knows the topic
• Select a consultant who is culturally competent and can communicate clearly with different stakeholders (e.g., program managers, front line staff, community members)
• Clarify the roles and responsibilities of the consultant and stakeholders
• Establish a workplan and timeline with deliverables for the consultant
• Meet regularly with the consultant to monitor progress
• Use the consultant to build internal capacity
26
![Page 27: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/27.jpg)
Take Home MessagesTake Home MessagesTake Home MessagesTake Home Messages
• Evaluation is only valuable if it is used
• Evaluation should be part of a program implementation
• All team members should be involved in evaluation efforts
• Planning is key
Evaluation is FUN!
27
![Page 28: Fundamentals of Evaluation for Public Health Programs of Evaluation for Public Health Programs.pdfFundamentals of Evaluation for Public Health Programs 1. 2. Objectives By the end](https://reader030.vdocument.in/reader030/viewer/2022040215/5ed04d5ad491622cfa6106ef/html5/thumbnails/28.jpg)
Thank you!Thank you!Thank you!Thank you!
28