Evaluating the impact of OVC programs: standardizing our methods
Jenifer Chapman, PhD
Senior OVC Advisor
MEASURE Evaluation
Overview
Background
Purpose of the OVC Survey Tools
Process of development
Guiding principles
Structure and content
When to use the Tools
When not to use the Tools
Using the data
Tools in a Toolbox
There is no single data collection tool that can meet all OVC program targeting, case management and M&E requirements.
This set of survey tools responds to distinct information needs related to program planning and evaluation, and fills a tools gap.
The problem
High investment in OVC programs BUT impact is unclear & questions regarding “what works” in improving household well-being
Part of the challenge: lack of standardized measures and tools for child and household outcomes (well-being)
4
A proposed solution
Standardized questionnaires for use in a survey of children ages 0-17 years and their adult caregivers
The purpose
Standardize population-level child and caregiver well-being data beyond what is available from routine surveys
Produce actionable data to inform programs and enable mid-course corrections
Enable comparative assessments of child and caregiver well-being and household economic status across a diverse set of interventions and regions
Who are these tools for?
Local and international research institutions and other implementing organizations with evaluation agenda
Our Process
Two step, participatory process:
Build consensus around core impact indicators for PEPFAR-funded OVC programs
Develop OVC program evaluation (survey) tools
Distilling the core indicators
Step 1: Extensive literature search
Step 2: Gaps (HES, PSS) filled through targeted research
Result: >600 child/HH wellbeing questions/indicators
Step 3: Analysis against 8 criteria
Result: shorter list of questions for discussion
9
Inclusion criteria
1. Measures impact/outcomes
2. Amenable to change from program interventions
3. Relevant across a wide range of interventions
4. Contributes to a holistic vision of child wellbeing
5. Verifiable through another source
6. Easy to implement
7. Relevant across different regions / countries
8. Relevant or easily adapted across age and sex
10
Finalizing the core indicators
External working group: solicited review from 49 stakeholders
Received feedback from > 25 individuals/groups
Finalized core set of 12 child and 3 household measures
11
From indicators to tools: Guiding principles
Questionnaires measure program outcomes
Program outcome data should be collected by trained data collectors
A documented protocol is required
Protocol with tools needs to undergo ethical approval both in the country of data collection and in the US
Tools require pilot testing in new settings before use
Developing the tools
Tools drafted with strong stakeholder input
Draft tools piloted in Zambia (and Nigeria) Cognitive interviews to
test key concepts (e.g. social support)
Household pre-test of full tools, procedures
Structure and content
1. Caregiver questionnaire (including questions on household)
2. Child questionnaire (ages 0-9 years), administered to caregiver
3. Child questionnaire (ages 10-17), administered to child with parental consent & child assent
Sections Core questions Optional modulesSection 1: Household schedule
Household schedule* (10) Changes in household
composition (4)
Section 2: Background Information on Caregiver and Household
Demographic information* (7) Work* (3) Access to money (3) Shelter (1)
Household Economic Status (forthcoming)
Progress out of Poverty Index or similar (country specific)
Section 3: Food Security Household food security (6) Dietary Diversity (1)
Section 4: Caregiver Well-being and Attitudes
General health (2) Caregiver support (4) Parental self-efficacy (1)
Perceptions and experience of child discipline, violent discipline (forthcoming)
Gender roles and decisionmaking power* (9)
Section 5: HIV/AIDS Testing, Knowledge, Attitudes
Basic HIV/AIDS knowledge* (7) HIV testing* (3) Attitudes to condom educ (1)
HIV/AIDS attitudes* (4)
Section 6: Access to HIV Prevention, Care & Support
Household access to services (1)
*DHS, bold=core indicator
Caregiver questionnaire
Sections Core questions Optional modulesSection 1: Child Health and Welfare
Confirm demographics (5) General health & disability (4) Birth certificate (2) Vaccinations (11) Fever (<5 years)* (1) Diarrhea (<5 years)* (1) Experience of neglect (2) Slept under mosquito net* (1) HIV testing experience* (2)
Fever: extended* (4) Diarrhea: extended* (3) Health for children
living with HIV/AIDS (forthcoming)
Section 2: Education and Work
School attendance*, progression/repeats, drop-outs, missed school days (5+ years) (9)
Work for wages (2) Early childhood stimulation (2)
Section 3: Food Consumption
Food consumption (2+ years) (8) • Dietary diversity (1)
Section 4: Access to HIV Prevention, Care & Support
Child access to services (1)
Section 5: Anthropometric Measures (of Children)
Weight*, Height*, MUAC
Child questionnaire (ages 0-9)
*DHS, bold=core indicator
Child questionnaire (ages 10-17)Sections Core questions Optional modulesSection 1: Background Information on Child
Confirm demographics* (5) Identity of caregiver (1)
Section 2: Diary Daily log (6) Section 3: Education School attendance*,
progression/repeats, drop-outs (9)
Section 4: Chores & Work Chores (3) Work (7)
Section 5: Food & Alcohol Consumption
Food consumption (8) Alcohol consumption (3)
Dietary diversity (1)
Section 6: Health, Support & Protection
Birth certificate (2) General health & disability (3) General support (4)
Health for children living with HIV/AIDS (forthcoming)
Perceptions/experience of violence (forthcoming)
Section 7: HIV Testing, Knowledge, and Attitudes
Basic HIV/AIDS knowledge* (7) HIV testing * (3)
Child development knowledge (6) HIV/AIDS attitudes and beliefs (4)
Section 8: Sexual Experience • Sexual behavior (13-17 yrs) (5)
Section 9: Access to HIV Prevention, Care & Support
Child access to services (1)
Section 10: Anthropometric Measures: Weight and Height
Weight, Height, MUAC
When are these the right tools?
Tools are useful if your question is:
1.Is my program having, or did my program have an impact on the children and households it reached?
2.What are the characteristics of children and their caregivers in my country, state/province or district/area, in terms of education, health, protection, and psychosocial support?
3.Where do the children most in need of program support live?
4.Approximately how many children need services or support?
5.What are the needs of my program’s registered beneficiaries, in terms of education, health, protection, and psychosocial support?
These are not the right tools for you if…
You want to know: Which children in selected communities to targetHow a particular child/household is faring Which households, children or caregivers are worst off What services to provide or refer for a particular child / householdThe number of children/households that are receiving program support, and the types of support receivedWhether staff are carrying out their responsibilities Whether interventions are being implemented as planned
And, why a special OVC survey?
DHS and MICS take a general population sample difficult to discern the program’s contribution
DHS and MICS include some, but not all of the OVC core indicators
Using the Data
Representative sample of program beneficiaries
Data collected at one point in time
Data collected at two points in time
Representative sample of the general population
Beneficiary sample: 1 point in time
Often called: Baseline, Midline or Endline
If baseline or midline: Use data for program planning or design, or mid-course corrections
Example: high food insecurity found at baseline
Result: Change in workplan, PMP agreed between partner and USG, emphasizing food security
If endline: Use data to inform follow-on activities
Beneficiary sample: 2 (or more) points in time
Commonly referred to as an “evaluation” Baseline data should be used immediately Evaluation results inform future programming, policy But, change in wellbeing from time 1 to time 2, does
not mean program is 100% responsible Much stronger result if:
Comparison group is added (counterfactual)
Panel study / cohort
General population sample
Commonly called a Situation Analysis
Use data for needs-based resource allocation at national or sub-national level (not individual or community level)
Example:
Nigeria OVC Situation Analysis
Triangulate
Analyze data alongside DHS and MICS data
Check for differences and similarities
Determine whether differences or similarities make sense, or point to a data quality issue
Implementation so far
Zambia: Impact evaluation of savings and internal lending communities on OVC wellbeing
Baseline data available early Fall
Nigeria: Baseline survey of OVC umbrella grant mechanism beneficiaries in 10 states (planning phase)
You said a toolkit?
Tools & Manual Data analysis guide Template protocol with
consent/assent forms Data collector training
materials And then what?
Revising as we learn
Supporting countries to implement
Where can I find out more?
Go to our website:http://www.cpc.unc.edu/measure/our-work/ovc
Keep in touch on Child Status Net:http://childstatus.net/
Email: Jenifer Chapman: [email protected] and Janet Shriberg: [email protected]
The research presented here has been supported by the President’s Emergency Plan for AIDS Relief (PEPFAR) through the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement GHA-A-00-08-00003-00. Views expressed are not necessarily those of PEPFAR, USAID or the United States government.
MEASURE Evaluation is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill in partnership with Futures Group, ICF International, John Snow, Inc., Management Sciences for Health, and Tulane University.
Questions?