evavluation of large scale health programs
DESCRIPTION
Planning the Evaluation Impact models Types of inference and choice of design Defining the indicators and obtaining the data Carrying out the evaluation Disseminating evaluation findings Working in large-scale evaluationsTRANSCRIPT
![Page 1: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/1.jpg)
EVALUATION OF LARGE SCALE HEALTH PROGRAMS
By: Adam F. Izzeldin; BPEH, MPH, PhD candidate.
Department of International Health, TMDU
CESAR G. VICTORA et., al. : evaluation of large scale health programs; Michael H. Merson, Robert E. Black, Anne J. Mills. Global health: Diseases, Programs, Systems and Policies, 2011
![Page 2: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/2.jpg)
ContentsContents
Planning the Evaluation
Impact models
Types of inference and choice of design
Defining the indicators and obtaining the data
Carrying out the evaluation
Disseminating evaluation findings
Working in large-scale evaluations
![Page 3: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/3.jpg)
Why We Need Large-Scale evaluation?
• In spite of large investments aimed at improving health outcomes in low- and middle-income countries, few programs been properly evaluated ("Evaluation," 2011; Evaluation Gap Working Group, 2006; Oxman et al., 2010).
• Each year billions of dollars are spent on thousands of programs to improve health, education and other social sector outcomes in the developing world, but very few programs benefit from studies that could determine whether or not they actually made a difference (Evaluation Gap Working Group, 2006).
![Page 4: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/4.jpg)
Types of evaluations
• External evaluation:
Independent
Carried out by researchers not involved in implementation
Funded by third party
• Internal evaluation:
Dependent
Carried out by implementing institutions
Funded by implementers themselves
• Two categories for evaluation: formative and summative.
![Page 5: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/5.jpg)
Examples for large scale evaluations
• The Multi-Country IMCI Evaluation
• Accelerated Child Survival and Development Initiative
• Tanzanian National Voucher Scheme for Insecticide-Treated Nets
![Page 6: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/6.jpg)
1. Planning the evaluation
•Who Will Carry Out the Evaluation?
•What Are the Evaluation Objectives?
•When to Plan the Evaluation?
•How Long Will the Evaluation Take?
•Where Will the Evaluation Be Carried Out?
![Page 7: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/7.jpg)
Who Will Carry Out the Evaluation?
• For internal evaluation: implementing institutions themselves or sometimes with the help of external consultants for specific tasks.
• For external evaluation: national or international institution of research will be recruited (UNICEF commissioned the Bloomberg School of Public Health at Johns Hopkins University to conducted an independent retrospective evaluation of ACSD in Benin, Ghana, and Mali)
![Page 8: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/8.jpg)
What Are the Evaluation Objectives?
• To review the available documentation on program objectives and goals, and to turn these items into evaluation objectives.
• The ultimate objective of an evaluation is to influence decisions.
• Funders interested in impact outcomes:
(Their decisions will be whether to continue funding ,or strategy needs to be reformulated)
• Local implementers interested in quality of service and population coverage:
(Their decisions are related to improving the program through specific actions)
![Page 9: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/9.jpg)
When to Plan the Evaluation?
• Before implementation; at the time the program is being designed
• Early onset, prospective evaluations allow collection of baseline data.
• Allows thorough, continuing documentation of program inputs and the contextual variables that may affect the program's impact.
• Early planning may enable the evaluation team to influence how the program is rolled out, thereby improving the validity of future comparisons.
• A disadvantage of prospective evaluations is that program implementation may change over time for reasons that are outside the control.
![Page 10: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/10.jpg)
How Long Will the Evaluation Take?
• The answer depends on whether the evaluation is retrospective, prospective, or a mixture of both techniques
Fully prospective evaluations include sequential steps:1. Collect baseline information2. Wait until the large-scale program is fully
implemented and reaches high population coverage
3. Allow time for a biological effect to take place in participating individuals
4. Wait until such effect can be measured in an endline survey
5. Clean the data and conduct the analysis
![Page 11: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/11.jpg)
Where Will the Evaluation Be Carried Out?
• Many large-scale programs are implemented simultaneously in more than one country
• This decision is usually taken in agreement with the implementation agencies
• Selection criteria should include characteristics that are desirable in all participating countries (geography, health system strength, and epidemiological profiles, and health system etc.)
• The rationale for selecting some countries and not others, because will affect the external validity or generalizability of the evaluation findings
![Page 12: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/12.jpg)
2. Developing an Impact Model….
• The model helps to clarify the expectations of program planners and implementers
• Contributes to the development of the evaluation proposal
• Helps guide the analyses and attribution of the results
• Can help track changes in assumptions as these evolve in response to early evaluation findings.
• Helps implementers and evaluators stay honest about what was expected
![Page 13: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/13.jpg)
health services attendance rates or mosquito nets
health services attendance rates or mosquito nets
Diagram
training, logistics, and management
training, logistics, and management
Inputs Inputs Outputs Outputs Process Process
staff, drugs, Equipment, teaching materials
staff, drugs, Equipment, teaching materials
percentage of women giving birth at a healthcare facility or the proportion of children sleeping under an insecticide-treated mosquito net
percentage of women giving birth at a healthcare facility or the proportion of children sleeping under an insecticide-treated mosquito net
Outcomes Outcomes
Common framework for evaluation
impacts impacts
reduced mortality or improved nutrition
reduced mortality or improved nutrition
![Page 14: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/14.jpg)
The IMCI Impact Model
Training ofhealth
workers
Family and community interventions
Health system improvements
Improvedhousehold
compliance/care
Improvedcareseeking &
utilization
Improved preventive practices
Introduction of IMCI
Improved quality of care in health facilities
Improved health/nutrition & reduced mortality
Increased coverage for curative & preventive interventions
![Page 15: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/15.jpg)
Development of an Impact Model
Steps in the Development of an Impact Model
Step Details
Learn about the program
• Read documents• Interview planners and implementers• Carry out field visits• Use special techniques as needed: cards, sorting exercise
Develop drafts of the model
• Focus on intentions and assumptions• Document responses from implementers• Record iterations and changes as model develops
Quantify and check assumptions
• Review existing evidence and literature • Identify early results from the evaluation- Documentations: what was actually done?- Outcomes: are assumptions confirmed?
Use and evaluate the model
• Develop an evaluation design, testing each assumption if possible
• Plan for analysis, including contextual factors• Analyze• Interpret results with participation by implementers
![Page 16: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/16.jpg)
A Stepwise Approach to Impact Evaluations
1.Policies; results-based planning: Are the interventions and plans for delivery technically sound and appropriate for the epidemiological and health system context?
2.Provision: Are adequate services being provided? at health facility/community levels?
3.Utilization: Are these services being used by the population?
4. Effective coverage: Have adequate levels of effective coverage been reached in the population?
5.Impact:Is there an impact on health and nutrition?
6.Cost-effectiveness: Is the program cost-effective?
![Page 17: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/17.jpg)
3.Types of inference and choice of design
• Adequacy Evaluations (converge)
• Plausibility Evaluations (comparison group)
• Before-and-After Study in Program and Comparison Areas
• The Ecological Dose-Response Design
• Randomized (Probability) Evaluation Designs
• Stepped Wedge DesignFigure 16-3 Simplified Conceptual Framework of Factors Affecting Health, from the Standpoint of Evaluation Design
Impact
![Page 18: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/18.jpg)
4. Defining the indicators and obtaining the data
• Documentation of Program Implementation
• Measuring Coverage (house hold surveys)
• Measuring or Modeling Impact
• Describing Contextual Factors
• Measuring Costs ( unit cost, operations, utilizations)
• Patient-Level Costs (severity of illness )
• Facility-Level Characteristics (quality, scope of service )
• Contextual Variables ( transport, supervision, pa tients' ability to access care)
• Data Collection Methods (cost) and Allocation Methods
![Page 19: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/19.jpg)
5. Carrying Out the Evaluation
• Starting the evaluation clock
• Feedback to implementers and midstream
corrections
• Linking the independent evaluation to routine
monitoring and evaluation
• Data Analyses
• Analyzing Costs and Cost-Effectiveness (process,
intermediate, and outcome in dicators)
• Interpretation and Attribution
![Page 20: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/20.jpg)
Types of process, intermediate, and outcome indicators and data needed Type of Indicator Indicator What measured Additional data
Process cost-effectiveness Expected costs and value for money
Budget projections, work plans, coverage
Process total cost perperson treated
Services provided Utilization rates
Process total cost perpreventive item
Services provided Utilization rates
Process Cost per capita Services provided, program effort
Population
Intermediate cost of qualityimprovement
Treatment leading to health gains
Utilization rates adjusted by quality
Outcome cost per deathaverted
Mortality reduction Mortality rates
Outcome cost per lifeyear gained
Mortality reduction Mortality rates and age of death (and life expectancy
![Page 21: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/21.jpg)
Joint interpretation of findings from adequacy and plausibility analysis How did program areas fare relative to nonprogram areas? (plausibility assessment)
How did impact indicators change over time in the program areas (adequacy Assessment)
Improved No change Worsened
Better Both areas improved, but the program led to faster improvement
Program provided a safety net
Program provided a partial safety net
Same Both areas improved; no ev idence of an additional program impact
No change in either area; no evidence of program impact
Indicators worsened in both areas; no evidence of a safety net
Worse Both areas improved; presence of the program may have precluded the deploy ment of more effective strategy
Program precluded progress; presence of the program may have hindered the deploy ment of more effective strategies
Program was detrimen tal; presence of the program may have hin dered the deployment of more effective strategies
![Page 22: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/22.jpg)
6. Disseminating Evaluation Findings and Promoting Their Uptake
• Policy makers and
program
implementers at
country level.
• Global scientific
public health
communities.
![Page 23: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/23.jpg)
7.Working in Large-Scale Evaluations
• First, good evaluations require effective communications
• Second, good evaluations require a broad range of skills and techniques, as well as an interdisciplinary approach.
• Third, good evaluations require patience and flexibility.
![Page 24: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/24.jpg)
8. Conclusion
• Conducting large-scale evaluations is not for the
fainthearted. This chapter has focused on the
technical aspects of designing and conducting an
evaluation, mentioning only in passing some of
the political and personal challenges involved
![Page 25: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/25.jpg)
Message taken home
• Ideal designs (based on text books like
this one) must often be modified to
reflect what is possible and affordable
in specific country contexts.
![Page 26: Evavluation of large scale health programs](https://reader033.vdocument.in/reader033/viewer/2022051412/549ea1bdb37959664b8b4661/html5/thumbnails/26.jpg)
Thank you for listening