the hwts m&e toolkit: work-to-date and way...
TRANSCRIPT
Daniele Lantagne, PE
The HWTS M&E Toolkit:
Work-to-date and Way Forward
Icddr,b The Larger Picture
Evaluation HWT Options
“provides a basis for which to
assess the microbiological
performance of HWT options”
Using QMRA analysis with three
target pathogens
- Campylobacer jejuni
- Cryptosporidium
- Rotavirus
Audience
- developers, manufacturers
- regulatory authorities
- certification organizations
Origins of the M&E Toolkit
Initially, an Aquaya tool-kit on health impact analysis
Health impact Important, critical, necessary
Outside the ability of most M&E staff
Monitoring is the routine assessment of a program’s activities Primary objective of measuring whether activities are carried out
as planned.
Evaluation is the systematic assessment of whether a program
has achieved its proposed objective
Outputs/outcomes are more easily measured than impact
Icddr,b HWTS M&E Framework
Icddr,b Need for an M&E Toolkit
Not meant to replace Evaluating HWT Options, meant to supplement
Encourage standardization and quality in M&E
Overcome perception health outcomes are necessary, best, only
0
1000
2000
3000
4000
5000
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21
Intervention Week
Nu
mb
er
Fa
mil
ies
0
20
40
60
80
100
% F
am
ilie
s
Number FamiliesPicking Up Product
Percent FamiliesPicking up Product
Icddr,b Need for an M&E Toolkit
Not meant to replace Evaluating HWT Options, meant to supplement
Encourage standardization and quality in M&E
Overcome perception health outcomes are necessary, best, only
Baseline Week 1 Week 2 Week 3 Week 4
# children/HH surveyed 49 48 40 29 12
# children suffered diarrhea
20 5 1 1 0
% children with diarrhea of total surveys
41% 10% 3% 3% 0%
The results suggest that [HWT option] has made a significant difference.
Currently, there is a lack of standardized, cross-cutting tools to
assist implementers, donors, and decision-makers in evaluating HWTS programs. This document seeks to fill this void by
providing practical recommendations for monitoring outputs
and outcomes of HWTS programs and explanations of the
more technical requirements for assessing impact.
Draft content
1: Overview of HWTS
2: Key aspects of evaluating HWTS
3: Gaps in understanding of health benefits
4: Essential and expanded indicators
5: Conducting field evaluation
6: Basic data analysis and dissemination
Annexes (health impact, etc)
Objective and Content
Icddr,b Responses to draft
1) THANK YOU to everyone who commented!
2) 19 pages of comments / reaction
3) Need for revision
Overall comments
Expand basis for conducting M&E
Revise indicators
Provide actual tools
Condense and simplify text
Broaden scope to consider M&E systems
Sustainability and M&E
Icddr,b Reframing - Context
Introduction Intended audience, scope
HWTS Options, WSP, critiques & strengths
M&E Outputs/outcomes/impact (do-able actions)
Linking to international and national policy
Indicators Basic, expanded, integrated, 1-5-15-full survey
Monitoring Internal, ongoing, normal visits, longitudinal
Evaluation External, point-evaluation, cross-sectional
Analysis Data, methods, descriptive epi, Chi-squared
Reporting How to disseminate, influence future programs
Icddr,b Reframing - Annexes
Annex A: Sample monitoring surveys
Annex B: Sample evaluation surveys
Annex C: Enumerator training and supervision
Annex D: WQ Testing - Chlorine Residual (CDC?)
Annex E: WQ Testing - Microbiological (CDC?)
Annex F: Sample statistical methods for analysis
Annex G: Sample monitoring report
Annex H: Sample evaluation report
CD? Supplementary materials? Downloadable? Annex?
Icddr,b Drilling down - summary data
Drilling down - observation
Icddr,b Drilling down - hard outcomes
Hard outcomes
Chlorine residual
Filter wet, water present
Microbiology
Results
84% (16/19) of HH use
contaminated water
47% (9/19) of HH
effectively treat
16% (3/19) don’t need
42% (8/19) not effective
Gets at: Was this intervention actually needed in this context? IMPACT.
Icddr,b Next steps
Discuss here the overarching document
NOT “specific of indicators”, please
Content, framework, TOC, tone, audience?
How far to go into impact vs. outcomes?
Can we agree on a framework?
Develop a 1-pager on this framework
Revise the document
Over the next few months
Release for another 1-2 rounds of edits
Here really dig into the details
???
Icddr,b The Larger Picture - Haiti?
The HWTS sub-cluster in Haiti
Developing certification for HWTS products
Should we verify using new WHO doc?
Current lab capacity
DINEPA Working on bacterial testing
MSPP High-quality lab (CDC), low other
NGO’s Bacterial testing
Low capacity to test crypto, rota, campy
WHO asks - what should we invest in?
Icddr,b This document…
Will hopefully NOT:
Provide guidance for experts, researchers, NSF or EPA regulators
Detail how to show efficacy of an HWTS option
Prescribe exactly how to complete M&E
Will hopefully:
Provide guidance for governments, NGOs, field evaluators
Provide information and examples (tool-kit)
Provide a do-able M&E strategy
Detail how to document effectiveness of HWTS option in specific contexts
Icddr,b The Larger Picture - Cohesion
These two documents act in concert
Dependent on local capacity
- Haiti / Cambodia / emergencies
Verify efficacy of HWTS products
Evaluating HWT Options
In collaboration with int’l researchers
Verify local effectiveness
Monitoring & Evaluation
In collaboration with local evaluators
Thank you!