![Page 1: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/1.jpg)
UNIVERSITY OF WASHINGTON
Accountability, transparency and corruption
in global health: the critical role of health
metrics and evaluation
Stephen S Lim
Assistant Professor of Global Health
![Page 2: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/2.jpg)
2
Outline
• Context
• Two examples• Tracking immunization coverage
• Conditional cash transfers to women for delivering in a health facility
![Page 3: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/3.jpg)
3
Outline
• Context
• Two examples• Tracking immunization coverage
• Conditional cash transfers to women for delivering in a health facility
![Page 4: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/4.jpg)
4
Development Assistance (Billions US$) for Health by Institution, 1990, 2007
![Page 5: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/5.jpg)
5
Global goals, funders and initiativesGoals• 1978: Alma-Ata: Primary Health Care and Health for All• 1984: Universal childhood immunization by 1990• 2000: Millennium Development Goals• 2003: 3 by 5 – 3 million people on antiretroviral drugs by 2005• 2008: Malaria Elimination
Private philanthropic organizations• 2000: Bill and Melinda Gates Foundation• 2006: Warren Buffet pledged additional US$30 to BMGF
Global Health Initiatives• 2000: Global Alliance for Vaccines and Immunizations (GAVI)• 2002: Global Fund for Aids, Tuberculosis and Malaria (GFATM)• 2003: US President’s Emergency Plan for AIDS Relief (PEPFAR)• 2005: US President’s Malaria Initiative (PMI)• 2007: International Health Partnership (IHP+)
![Page 6: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/6.jpg)
6
National health sector reforms and programs
2001: Thailand Universal Coverage
• 30 Baht Scheme
2003/04: Mexico Health Sector Reforms
• System of Social Protection in Health including Seguro Popular
2005: Indian National Rural Health Mission
• Conditional cash transfers for women to give birth in health facilities
![Page 7: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/7.jpg)
7
Critical role of health metrics and evaluation
• High-quality measurement of health indicators and evaluation of programs is central to issues of transparency and accountability
• Are resources being used effectively?
• Have initiatives and reforms led to improvements in health system delivery and population health?
• Mounting pressure from funders, civil society, etc to document this
• Economic crisis has led to rising fears that development assistance budgets will be cut
![Page 8: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/8.jpg)
8
Outline
• Context
• Two examples• Tracking immunization coverage
• Conditional cash transfers to women for delivering in a health facility
![Page 9: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/9.jpg)
9
Tracking childhood immunization coverage
• Substantial resources have been invested in delivering immunization services
• Basic vaccines, e.g. three-dose diptheria, pertussis and tetanus (DTP3) as well as new vaccines, e.g. HiB, rotavirus
• Global initiatives• 1984: Universal Childhood Immunization (UCI) by 1990 initiative, defined as 80%
immunization coverage
• 1999: Global Alliance on Vaccines and Immunizations (GAVI)
• Monitoring the extent of immunization delivery is critical for evaluating how effective these investments and initiatives have been
![Page 10: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/10.jpg)
10
Tracking childhood immunization coverage
• GAVI’s Immunization Services Support (ISS) is the funding that aims to increase coverage of basic vaccines such as three-dose diptheria, tetanus and pertussis (DTP3) vaccination.
• ISS payments are performance-based with funds disbursed in proportion to the number of additional children targeted or reported to receive DTP3.
• Number of additional children receiving DTP3 is based on official reports from countries to WHO and UNICEF.
![Page 11: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/11.jpg)
11
Tracking childhood immunization coverage
Two main questions:
1. What is the trend in the fraction of children receiving three-dose diptheria, tetanus and pertussis vaccination (DTP3 coverage) over the period 1986 to 2006?
2. Do target-oriented initiatives such as universal childhood immunization (UCI) and results-based financing initiatives such as GAVI’s Immunization Services Support (ISS) lead to over-reporting of DTP3 immunization coverage?
![Page 12: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/12.jpg)
12
Data sources• Micro-data from standardized multi-country surveys
• DHS, MICS, CDC• Crude coverage: three DTP vaccinations by maternal recall or card documented,
irrespective of vaccine schedule• Estimated for each birth cohort (up to 5 years prior to the survey)• 225 surveys
• Survey reports and WHO/UNICEF database:• 78 multi-country surveys• 142 country-specific surveys with sample size reported• 145 country-specific surveys without sample size reported
• Administrative data estimates based on health service provider registries• Reported routinely to WHO and UNICEF since 1990• Officially reported estimates since 1980
![Page 13: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/13.jpg)
13
Quick review of Immunization Services Support (ISS)
• Performance-based payment• Number of additional children reported by countries to have received DTP3
• Reports largely based on administrative data
• Baseline is the year prior to approval of the proposal
• US$20 is paid once per additional child
• Data quality audit (DQA) of administrative data system before reward payments commence (from Year 3)
![Page 14: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/14.jpg)
14
![Page 15: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/15.jpg)
15
![Page 16: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/16.jpg)
16
![Page 17: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/17.jpg)
17
![Page 18: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/18.jpg)
18
Estimating missing survey coverage
Two purposes:
1. Generate plausible estimates over time to allow monitoring of indicators
2. Reduce compositional bias in in causal inference that can stem from missing data
We use validated statistical approaches that are
• Objective
• Replicable
• Characterizes uncertainty
![Page 19: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/19.jpg)
19
![Page 20: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/20.jpg)
20
Global trends in DTP3 coverage
Survey-based global coverage of DTP3 (black) with 95% uncertainty estimates compared to countries’ officially reported (red) and WHO and UNICEF estimates (blue), 1986 to 2006.
![Page 21: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/21.jpg)
21
Regional trends
![Page 22: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/22.jpg)
22
Does ISS lead to over-reporting?
• Statistical analysis of over-reporting (officially reported coverage minus survey coverage) by years since the GAVI ISS baseline
-0.05
0.00
0.05
0.10
0.15
0.20
0.25
0.30
0.35
-1 0 1 2 3 4 5 6 7 8
Years post-GAVI ISS baseline
Bet
a-co
effi
cien
t
BDDR - PCSE
BDDR - ABBB
Multiple Imputation - PCSE
Multiple Imputation - ABBB
![Page 23: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/23.jpg)
23
GAVI Immunization Services Support (ISS)• Number of additional children vaccinated in 51 countries receiving ISS
funding up to the year 2006 :• Based on official reports: 13.9 million• Survey-based: 7.3 (5.5 to 9.2) million
• ISS payments• Based on official reports: US$289 million• Survey-based: US$148 million
• Over-reporting is not uniform• 4 countries that reported increases, number of additional children did not increase• 6 overestimated by > 4x• 10 overestimated by > 2x but ≤ 4x• 23 overestimated by > 1x but ≤ 2x• 8 countries underestimated
![Page 24: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/24.jpg)
24
Implications
• At the global level, survey-based immunization coverage has increased continuously and gradually over the last 20 years• Reflects time and investment needed to expand health services
• Improvements more pronounced in some regions (e.g. Central, West sub-Saharan Africa) and countries (e.g. Cameroon) during recent time period
• Targets and payments for performance such as GAVI’s ISS can incentivize improvements but also lead to over-reporting
• Over-reporting likely reflects pressures throughout the reporting system to meet targets
• Monitoring and evaluation systems need to be based on independent, rigorous, empirical measurements that are robust to these effects
![Page 25: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/25.jpg)
25
Outline
• Context
• Two examples• Tracking immunization coverage
• Conditional cash transfers to women for delivering in a health facility
![Page 26: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/26.jpg)
26
Janani Suraksha Yojana – “Safe motherhood scheme”
• Launched in 2005; 100% centrally funded
• Goal: reduce maternal and neonatal mortality
• Works by incentivizing women to deliver in a health facility
• Implemented through Accredited Social Health Activists (ASHAs)
• ASHAs also receive a cash benefit
• Budget allocation of US$342 million in 2009-10
Accredited Social Health Activist (ASHA)Madhya Pradesh, IndiaPhoto: Department for International Development, UKAID
![Page 27: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/27.jpg)
27
National guidelines, Eligibility
In 10 high-focus states • All pregnant women delivering in government
facility or accredited private institutions
Other states & home deliveries• Below the Poverty Line
• >19 years of age
• First two live births
• Targeted to women from scheduled caste or tribe
![Page 28: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/28.jpg)
28
National guidelines, Cash payments
Rural Urban
Mothers ASHA Mothers ASHA
High-focus states 1,400 600 1,000 200
Other states 700 600 600 200
Home deliveries 500 500
1 U.S. dollar ~ 45 Indian Rupees
![Page 29: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/29.jpg)
29
Questions
• What is the level of implementation of JSY at district and state-levels?
• Is JSY reaching its intended beneficiaries?
• Does receipt of financial assistance under JSY lead to increased antenatal care and in-facility delivery and reduced perinatal, neonatal and maternal mortality?
![Page 30: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/30.jpg)
30
Data
• India District-level Household Surveys (DLHS)• DLHS-2: ~1,000 households from 593 districts, 2002 to 2004
• DLHS-3: 1,000 to 1,500 households from 611 districts, late 2007 to early 2009
• Ever-married women aged 15 to 44, for most recent pregnancy• Antenatal care (no. of visits)
• Delivery care (type of provider, location)
• Outcome (live birth, still birth, spontaneous or induced abortion)
• Survival of the child in the case of a live birth
• Receipt of financial assistance under JSY (DLHS-3)
• Individual and household characteristics, e.g. asset-based wealth, caste, education, location of residence and distance to facility
![Page 31: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/31.jpg)
31
In-facility birth coverage, 2001 to 2003 Births receiving JSY, 2007/08
![Page 32: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/32.jpg)
32
JSY uptake by socioeconomic indicators, national-level
02468
1012141618
Urban Rural, 0-4 Rural, 5-9 Rural, 10-19 Rural, 20+
% JS
Y
Residence, distance from facility (km)
02468
1012141618
no education < primary < secondary ≥ secondary
% JS
Y
Maternal Education
02468
1012141618
sched. caste sched. tribe other caste/tribe
other
% JS
Y
Caste/Tribe
02468
1012141618
1 2 3 to 5 5 or more
% JS
YNumber of Births
02468
1012141618
1 (Poorest) 2 3 4 5 (Richest)
% JS
Y
Wealth Quintile
02468
1012141618
15-19 20-24 25-29 30-24 35-39 40-44
% JS
Y
Maternal Age (years)
![Page 33: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/33.jpg)
33
In-facility birth coverage, 2001-2003 In-facility birth coverage, 2007/08
![Page 34: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/34.jpg)
34
Evaluating impact of JSY on coverage and outcomes
• Exact matching• Match births receiving JSY to those not receiving JSY in DLHS-3
• Matching covariates: urban/rural residence, BPL card ownership, wealth quintile, caste, education, parity, and maternal age
• Logistic regression on matched data allows more precise control for confounders
• With-vs-without• Logistic regression, comparing births receiving JSY to births that did not receive
JSY in DLHS-3 and all births in DLHS-2
• District-level differences-in-differences• Compare districts by level of JSY uptake, controlling for baseline differences
(DLHS-2)
• 580 district aggregates from DLHS-2 to DLHS-3.
![Page 35: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/35.jpg)
35
Outcomes
Intervention coverage
• Antenatal care with at least three visits
• In-facility birth
• Skilled birth attendance (in-facility birth or birth outside of a facility with a skilled attendant)
Mortality
• Perinatal death (stillbirth or death up to and including 7 days after a live birth)
• Neonatal death (death up to and including 1 month after a live birth)
• Maternal mortality* (death of women aged 15 to 49 during pregnancy or up to 6 weeks after birth or termination)
* In district-level analysis only
![Page 36: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/36.jpg)
36
Potential confounders
Controlled for:
• maternal age;
• number of live births;
• birth interval;
• single or multiple birth;
• maternal education;
• household wealth based on asset ownership;
• caste/tribe;
• religion; and
• location of residence with respect to distance to the nearest health facility
• Varied using district, state-level fixed and random effects
![Page 37: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/37.jpg)
JSY and intervention coverage, national level
ANC, 3 visits In-facility birth Skilled birth attendance*
∆ Probability (95% CI)
∆ Probability(95% CI)
∆ Probability(95% CI)
District Diff-in-Diff
0.110 (0.047, 0.172)
0.492 (0.432, 0.551)
0.413 (0.352, 0.474)
Matching 0.111 (0.101, 0.122)
0.475 (0.468, 0.482)
0.400 (0.394, 0.406)
With-vs-without
0.109 (0.100, 0.119)
0.438 (0.431, 0.444)
0.356 (0.350, 0.362)
Change in probability of receiving intervention: JSY vs no JSY
![Page 38: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/38.jpg)
Impact on mortality, national level
Perinatal mortal(per 1,000 live
births)
Neonatal mortality(per 1,000 live
births)
Maternal mortality(per 100,000 live
births)
∆ probability(95% CI)
∆ probability(95% CI)
∆ probability(95% CI)
District Diff-in-Diff
-13.2(-30.1, 3.7)
-5.4 (-19.8, 8.9)
105.0(-247.3, 457.2)
Matching -3.6(-5.2, -2.1)
-2.2(-3.7, -0.8)
NA
With-vs-without
-4.1(-5.6, -2.5)
-2.4(-4.1, -0.8)
NA
Change in probability of death: JSY vs no JSY
![Page 39: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/39.jpg)
Variation by State: Intervention coverage
ANC, 3 visits In-facility birth Skilled birth attendance
∆ probability(95% CI)
∆ probability(95% CI)
∆ probability(95% CI)
High-focus states 0.108(0.096, 0.120)
0.629(0.621, 0.637)
0.573(0.566, 0.580)
Northeastern states
0.137(0.102, 0.172)
0.353(0.320, 0.386)
0.307(0.277, 0.337)
Other states 0.036(0.022, 0.049)
0.068(0.054, 0.083)
0.051(0.339, 0.063)
![Page 40: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/40.jpg)
Variation by State: Mortality
Perinatal mortality(per 1,000 live
births)
Neonatal mortality(per 1,000 live
births)
∆ probability(95% CI)
∆ probability(95% CI)
High-focus states -2.7(-5.0, -0.4)
-1.4(-3.5, 0.7)
Northeastern states -2.4(-6.1, 1.3)
-0.1(-4.0, 3.8)
Other states -5.2(-7.4, -3.0)
-4.1(-6.1, -2.1)
![Page 41: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/41.jpg)
41
Implications
• Varied uptake of JSY across states; not reaching the very poor
• Increases in ANC coverage and intra-partum care coverage
• Likely reductions in perinatal and stillbirth/neonatal mortality
• But potential quality of care issues in high-focus states indicates
• Alternative monitoring approach needed for maternal mortality
• Continued monitoring and evaluating the program is critical
![Page 42: Understanding the Effect of the GAVI Initiative on Reported Vaccination Coverage Levels](https://reader033.vdocument.in/reader033/viewer/2022061218/5481ae3fb07959650c8b45bb/html5/thumbnails/42.jpg)
42
Summary
• Substantial resources are being directed towards improving population health
• Need to track in a valid, reliable and comparable way health indicators and evaluate the impact of programs
• Ensure that increased resources for health are being utilized intended purpose and are making a difference to the health of populations
• Increasing relevance during a time of global financial crisis
• Independent and empirically-based monitoring of health indicators and evaluation of programs