pepfar ovc evaluation: how good at doing good? (gh tech/usaid 2011)
DESCRIPTION
PEPFAR OVC Evaluation: How good at doing good? (GH TECH/USAID 2011). Prof Lorraine Sherr UCL London UK Acknowledgements to Miriam Zoll (Co-author) , Natasha Croome , Adele Daniels and Jaya Karira. Task: Summarize and Analyze PEPFAR OVC Evaluations. 18 Evaluations 22 Programs - PowerPoint PPT PresentationTRANSCRIPT
PEPFAR
AIDS 2012 - Turning the Tide Together
PEPFAR OVC Evaluation: How good at doing good? (GH TECH/USAID 2011)Prof Lorraine Sherr UCL London UK Acknowledgements to Miriam Zoll (Co-author), Natasha Croome, Adele Daniels and Jaya Karira
Task: Summarize and Analyze PEPFAR OVC Evaluations
• 18 Evaluations• 22 Programs• 9 Countries – Kenya, Tanzania, Uganda, Mozambique, Rwanda, Namibia,
Zambia, South Africa and Haiti• 8 basic needs categories: Food/Nutrition, Health, Education, PSS, Economic
Strengthening, Child Protection, Shelter and HIV Prevention• Critique methodology and summarised findings.• http://resources.ghtechproject.net/content/pepfar-ovc-evaluation-how-good-doing-good
Evaluation – is it ever expedient?
1. Systematic reading by reviewers.2. Coding - extract themes agreed by
two reviewers. 3. Sub-analysis of costing.4. Methodology adequacy criteria.5. Seven basic needs categories. 6. Recommendations for future
evaluation and evidence- based policy.
Variable 1 2 Agree- ment
Refusal Rates 2 2 100%
Random Selection
2 3 66.7%
Gender 4 4 100%
Significance 4 4 100%
Comparison Group
3 3 100%
Cross Sectional 5 4 66.7%
Descriptive Data 7 7 100%
Double coding 6 reviews (Kappa =.83)
Evaluations Surveyed1. Brewster, Lee, et al, Catholic Relief Servies' OVC PEPFAR Programs: A Midterm Multicountry
Program Evaluation2. Nyangara, F., Obiero, W., Kalungwa, Z., Thurman, T., Community-based Psychological
Intervention for HIV-Affected Children and their Caregivers: Evaluation of the Salvation Army's Mama Mkubwa Program in Tanzania, USAID, AXIOS, Measure Evaluation, March 2009
3. Thurman, T., Hutchinson, P., Ikamari, L., Gichuhi, W., Murungaru, K., Nyangara, Community Education and Sensitization as an OVC Care and Support Strategy: Evaluation of the Integrated AIDS Program-Thika in Kenya, USAID, University of Nairobi Population Studies and Research Institute, Measure Evaluation, March 2009.
4. Thurman, T., Rice, J., Ikamari, L., Jarabi, B., Mutuku, A., Nyangara, F., An Evaluation: the Difference Interventions for Guardians Can Make: Evaluation of the Kilifi Orphans and Vulnerable Children Project in Kenya, USAID, Measure Evaluation, University of Nairobi Population Studies and Research Institute, March 2009.
5. Nyangara, F., Kalungwa, Z., Obiero, W., Thurman, T., Chapman, J., Promoting a Home-based Program Model for Supporting Children Affected by HIV/AIDS: Evaluation of Tumaini Project in Iringa Region, Tanzania, USAID, AXIOS, Measure Evaluation, March 2009
Evaluations Surveyed6. Swarts, Brian, et al, Empowering Better Care: Report on Economic Strengthening
for OVC Caregivers in Uganda, Summary of Findings, The Salvation Army World Service Office, WORTH, USAID and PACT, April 2010
7. Hutchinson, P., Thurman, T., Analyzing the Cost Effectiveness of Interventions to Benefit OVC: Evidence from Kenya and Tanzania, USAID, Measure Evaluation, April 2009
8. Nyangara, F., Obiero, W., Effects of a Community-Focused Approach Supporting the Mosts Vulnerable Children: Evaluation of SAWAKA Jali Watoto Program in Kagera, Tanzania, USAID, Measure Evaluation, August 2009
9. Thurman, T., Hoffman, A., Chatterji, M., Brown, L., A Case Study: Kilifi Orphans and Vulnerable Children Project, Measure Evaluation and Catholic Relief Services with USAID, March 2007
10. Nyangara, F., Thurman, T., Hutchinson, P., Obiero, W., Effects of Programs Supporting OVC: Key Findings, Emerging Issues and Future Directions
Evaluations Surveyed 11. Murombedzi, Cornelius, Mugumya, Firminus, Opportunity International/Habitat for Humanity:
Evaluation of Sustainable Income and Housing for Orphans and Vulnerable Children in Africa, International NGO Training and Research Centre, August 2009
12. Larson, B., Hamazakaza, P., Costella, J., Laurence, S., Costing of OVC Service Delivery in South Africa and Zambia, Center for Global Health and Development and Health and Development Africa, USAID/Project SEARCH, September 10, 2010
13. Jaden Consultancies, Weaving the Safety Net: Mid-Term Evaluation, Final Report, Christian Children's Fund and USAID, March 2008
14. Jaden Consultancies, Weaving the Safety Net: Mid-Term Evaluation, Z Scores Christian Children's Fund and USAID, March 2008
15. Scott, N., Pfaff, M., Semrau, K., Leyenaar, J., Suojanen, A., Mugala, T., Simon, J., End of Project Evaluation Report: Faith-based Regional Initiatives for Orphans and Vulnerable Children (FABRIC), USAID/Project Search, boston University OVC-CARE Project, Family Health International, August 2010
16. Yates, DeeDee, Finding the Inflection Point in Health Education and Economic Strengthening: Final Evaluation of Project HOPE's Track 1 OVC Program Sustianable Strengthening of Families and Orphans and Vulnerable Children Mozambique and Namibia, Cooperative Agreement #: GPO-A-00-05-00019-00, May 31, 2010
17. Capps, J., Bukuluki, P., Davis, S., Swarts, B., Bailey, B., Wakubona, S., Sustainable Community Support for Orphans and Vulnerable Children in Tanzania and Uganda, Final Salvation Army (TSA) Uganda, June 2010.
18. RAPIDS Evaluation Final Report 2005–2009 Key Findings; Population Council, Africare & Rural Net Associates, July 2009, Revised 2010
Overview and data abstraction
73
42
112
82
718
123
0 5 10 15 20
Refusal
Random selection
Consecutive selection
Baseline data
Power calc
Cross section
Ethical approval
Comparison groups
Theory
Number of studies
PEPFAR com
HIV OVC
Age IssuesFocus on Adolescents
• 8-14 (6)• 0-4; 0-18 (2)• 5-11 (1)• 7-12; 7-15 (2)• 13-18; 13-17 (2)• 12-19; 12-17 (2)• N/A – (2 costing studies)• 3 with unclear age
Grading methodology
• STRONG: No study used a RCT design in any form none graded as "strong".
• ADEQUATE: Nine studies graded "adequate" -some form of before/after, case control, and post intervention comparison design.
• 1 qualitative (with good design)• Two costing studies graded as adequate - utilised
consistent audit/costing data. • WEAK: Six studies were graded as "weak" and
conclusions/ findings should be viewed with caution.
Quantitative vs qualitative methodology
The war is over
• 13 used/included quantitative• 11 used/included qualitative • Qualitative is no excuse for poor methodology.• If qualitative used, quality standards on how sampled, how
recorded, how analysed – coding and themes – not just the use of quotes (which is what a number did)
Complex interventions• Most programmes are complex – mechanisms
of impact difficult to align.
Critical Absence of Baseline Data
Data Handling
• Vast number of data points (15 Scott)• Simplistic reporting (failure to do complex analysis)• Percentages versus statistical tests• Power calculations
Theory• Role played (low)• Recourse to theory (eg knowledge does not equate to behaviour – why focus and
measure of knowledge eg 9 Nyangara looks at knowledge gaining, some look at attendance at knowledge imparting (not even if they got the knowledge).
• Generalisability• Understanding mechanisms
Two forms of Evaluation – Against Pepfar Goals or against Programme goals
Methodological Trends
• Baseline: PEPFAR wants evidence-based approaches but most evaluations had no baseline, which is the only way to monitor and track impact of dollars and labor invested.
• Quantitative: Several of the evaluations were able to tell us how many OVC received services, how many caregivers had taken out a loan and repaid it.
• Standardized Indicators – good methodological reasons for using validated indicators (See Tulane group for good example).
• Outcome - How do you operationalize efficacy? (eg Yates 16 – use attendance at Health education as the indicator, but what if you attended and slept through?
• Gender: many mention few analyze
• Sampling – good methodological reasons for using excellent sampling and
reporting refusal rates.• Staff training in each year were a relatively large share of program costs. If such
expenses are considered investments in human capital, the training would provide services to the project over more than one year. If there is substantial staff turnover, prior investments in training would not provide services to the project in the future
• Dose Issue - Exposure dose seems to be an emerging issue. Home visiting is not a standardised commodity. Need to harmonise concepts in description and quantity standards
Findings by Basic Needs: Food/Nutrition - 14 / 18
• 9/14 evaluations focused on food/nutrition found OVC households to be moderately or severely food insecure.
• Improvements shown, but questions raised about dependence and sustainability.
Health Care ––11/18• 8/11 studies demonstrated improved health
outcomes• Specifically notable when combined with economic
strengthening and high dose/follow through
Education–– 15/18• Enrollment effect• Supplies positive mental health effect.• 0 / 14 identified gender imbalances in school
enrollment, although one reported that more girls than boys miss more than two days of school.
• Cost – resentment• Ongoing funding need until government shoulders
this.
Psychosocial––13/18
• 10/ 13 home- visits clearly benefited OVC but not always guardians
• Negatives associated with infrequent visits, low meeting attendance at Kids' Clubs
• Targeting carers indirectly benefits OVC• Volunteers overwhelmed by caseloads and ill equipped to
provide HB PSS counseling, particularly to adolescents and caregivers.
Economic Strengthening–9/18
• PEPFAR's investments in local savings and loan schemes are proving to be effective in raising many OVC households out of poverty
• Economic strengthening intersects significantly in other core areas of basic need, including food/nutrition, health, education, psychosocial support and shelter. Thus, economic strengthening remains one of PEPFAR's most potent and effective strategies for improving the lives of OVC and their families
• Costing - Great variation for example – Kids Clubs: $1.96 - $14.08 – Home Visiting: $3.55 - $89.68 – Gardening $24.00 - $89.00 – Food $1.39 - $32.36 – Educational Support: $17.23 - $141.32
Child Protection––16/18• Measurable role home-based visits play in
reducing child abuse, stigma and social isolation • Psychosocial interventions targeting guardians
often benefited OVC as well • Children whose homes had been visited by a
volunteer > birth certificate (36% of all children had birth certificates)
• Home visitor training associated with improved outcome
Shelter––6/18
• Shelter was the least prioritized basic need in all evaluations.• Provision of shelter made significant positive
differences in multiple areas, including protection, psychosocial support and health
HIV Prevention––15/18• Children's HIV knowledge appeared to increase in 7 out of 13
evaluations.• Some interventions appeared to be more thorough than
others.• School-based HIV prevention activities were part of 5 out of
13 prevention initiatives and were estimated to be cost effective.
Summary• Quality Standards. Define PEPFAR’s quality standards for evaluations?
Methodology? Standardization? The very mention of "PEPFAR" should be synonymous with quality
• Costing - Great variation• Volunteers - costly cyclical patterns are troublesome to program continuity and
quality, heavy reliance, questions of quality.• Food 3 / 13 focused on food - found > 80% of household samples were severely
food insecure (86%; 86.3%; 88.5%). • Kids’ Clubs – Mixed findings. Often volunteers poor results on issues of stigma and
discrimination.• One Stop Shop - Harmonizing and integrating complex PEPFAR-funded
OVC programs can yield better and multiple outputs • An emergency response should be saved for emergencies • Evidence- based programming should be a priority - There was weak
evidence of gains and impacts in the five core areas, but with a number of caveats
The way forward• Updated PEPFAR OVC Guidance (2012) stipulates: “Allocate
at least 10% of program budgets to M&E”• PEPFAR supporting the development and implementation of:
– Methodology to track impact of child welfare and protection systems strengthening
– Standardized quantitative program evaluation tool to measure child and caregiver/household outcomes across countries at population level (Will be introduced in Nigeria, Mozambique)
Evidence-Based Policy?
How Good at Doing Good?Reviewing the PEPFAR OVC Factor
andPEPFAR HKID Portfolio Review: Children of the HIV/AIDS Epidemic
Reports available at:http://resources.ghtechproject.net