Slide 1

Futures Group and the World Bank Institute in collaboration with Abt Associates, OHanlon Health Consulting, University of California at San Francisco and Tropical Health LLP JANUARY 2014

Slide 2

MARKETS FOR HEALTH SESSION 11 What do We Really Need to Know? M&E and its Application to M4H Ruth Berg Futures Group Futures Group and the World Bank Institute in collaboration with Abt Associates, OHanlon Health Consulting, University of California at San Francisco and Tropical Health LLP

Slide 3

MARKETS FOR HEALTH OVERVIEW M&E FUNDAMENTALS M4H CONSIDERATIONS REAL WORLD EXAMPLES SUMMARY

Slide 4

MARKETS FOR HEALTH Evaluation Types (the Short List) Types of evaluation Answers the questionBasic requirements Process evaluation / monitoring Is the programme on track? and Is change occurring according to plan? Construct good indicators Valid accurately measures behavior Reliable minimizes measurement error Precise defined in clear terms Measureable - quantifiable Timely in step with the programme Performance evaluation Is change occurring according to plan? And Can we plausibly attribute change in outputs or outcomes to the intervention? Triangulate information Program records and monitoring data Data from secondary sources Expert interviews Stakeholder opinions Impact evaluation Can we rigorously attribute change in outputs or outcomes to the intervention? Construct a counterfactual - estimate what the outcome would have been without intervention Experimental designs Quasi experimental designs Non-experimental designs

Slide 5

Evaluation Purpose Spectrum Process Evaluation/ Monitoring Performance evaluation Impact evaluation Throughout implementation Before implementation & post implementation Learning and adapting programme Learning and decision making (scale/replication) Improving Proving Type Timing Use Internal team External team ( close collaboration with internal team) External team ( close collaboration with internal team) Distance Adapted from ITAD 2013 Learning and adapting Mid-project & end of project

Slide 6

MARKETS FOR HEALTH OVERVIEW M&E FUNDAMENTALS M4H CONSIDERATIONS REAL WORLD EXAMPLES RECOMMENDATIONS SUMMARY

Slide 7

An ITAD (2013) review of 14 M4P evaluations concluded: The nature of the [M4P] approachpresents a number of challenges for evaluation The M4P evaluationswere often weak in terms of Consideration of systemic, sustainable changes in market systems Triangulation practices Theories of change (often linear) Considerations for Market System Approaches

Slide 8

InputProcess Health Output Health Outcome Health Impact Why? Dynamic Effects of Market System Approaches at Odds with Traditional Linear Results Chains Traditional Results Chain Large-scale

Slide 9

Requires More Adaptation and Change During the Process Stage: Less is Known at the Theory of Change Stage M4P Component E M4P Component D M4P Component B M4P Component A M4P Component C Setting the Strategic Framework Understanding Market Systems Defining Sustainable Outcomes Facilitating Systemic Change Assessing Change Monitoring & Evaluation Vision & Rationale Identification & Research Monitoring & Evaluation Implementation & Adaptation Information & Feedback

Slide 10

Systems Approaches are Often Unpredictable So the Learning Process can Look More Like This Learning MonitoringStrategy Adapted from Preskill and Beer, 2012

Slide 11

InputProcess Health Output Health Outcome Health Impact Linear Logic Model vs. Systemic Logic Model Systemic Large-scale Sustainable Linear logic model Systemic logic model for M4H Adapted from ITAD 2013 M4H Intervention Market Change Health Output or Outcome Sustainable Large-scale Systemic Adaptive Process

Slide 12

Impact Trajectory for Impact is Likely to Be Non-Linear Non-systemic approach Impact Time Systemic approach Start of projectEnd of project Baseline Endline Post-endline Source: Adapted from DCED 2011

Slide 13

Explicitly monitor Systemic change Whether changes in rules or supporting functions occurred (yes/no) Sustainability Whether market system change continues after intervention without external assistance Whether health outputs/outcomes continue after intervention without external support at the same or improved levels (yes/no) Scale Whether intervention crowds-in of other actors Whether intervention leads to copying by other actors % of targeted population reached by the intervention Market system, outputs or outcomes after the project ends What Does this Mean for M4H M&E? Process evaluation/monitoring

Slide 14

Plausible attribution Usually best option because it allows Non-linear impact trajectory Adaptive approach to implementation Expected spill-over effects Rigorous attribution through experimental designs Faces important challenges for systems approaches They dont allow Spill-over effects Contamination Need to stick to your original intervention (no adaptation as you go) Possible at pilot stage if Beneficiaries can be randomly assigned to treatment & control groups (RCTs) Baseline equivalency can be established in key variables (quasi-experiments) What Does this Mean for M4H M&E? Evaluation & Attribution

Slide 15

MARKETS FOR HEALTH OVERVIEW M&E FUNDAMENTALS M4H CONSIDERATIONS REAL WORLD EXAMPLES SUMMARY

Slide 16

MARKETS FOR HEALTH Using Plausible Attribution to Measure M4H Light in Morocco

Slide 17

R&D Rules Regulations Standards Laws Informal rules & norms SD Information Quality Assurance Subsidy Infrastructure Purchase Partnerships Health services & products Markets for Health Framework (M4H) Related Services Invest Private Sector Representative bodies Government Providers, Sellers Consumers, Patients Not-for-profits

Slide 18

Ministry of Healths system goal: Change incentives of OC manufacturers in enduring way to Ensure commercial sector reaches low & middle income MoH developed four-way OC partnership (1992) With USAID, Wyeth Pharmaceuticals, & Schering, & IPPF affiliate In consultation with Pharmaceutical Association The negotiation MOH Changed regulation to permit OC advertising Required brands to have common logo: Kinat al Hilal Wyeth and Schering Price brands 30% lower than nearest commercial product Nation-wide to achieve scale USAID Manage advertising fund & campaign during start-up Transition to IPPF affiliate to sustain campaign post-graduation Supporting Function Oral Contraceptive (OC) Partnership Morocco (1992 2003)

Slide 19

How did it work and how do we know (checking and adapting) Performance evaluation showed plausible attribution using Triangulation of different data sources Monitoring data from four rounds of Demographic and Health Surveys Pricing data Examination of different variables Do they tell a story consistent with the strategy and activities? From launch to post-project period OC use increased commercial share increased reduced prices remained stable equity improved How do we know if it worked?

Slide 20

% of MWRA Who Use OCs OC Demand Steadily Increased Percent Sources: Demographic and Health Surveys 1992, 1995, 2003: Commercial Market Strategy Survey: 2000

Slide 21

Commercial Share of OC Use Increased 19922003 Public & NGO Commercial OC Source Mix

Slide 22

72% 118% Commercial Share of OC Use by Wealth Quintile 18% Commercial Sector Reach among Poor Increased Percent

Slide 23

Current modern contraceptive use among women in union by wealth quintile, Morocco, 1987 2003. Agha S, and Do M Health Policy Plan. 2008;23:465-475 Published by Oxford University Press in association with The London School of Hygiene and Tropical Medicine The Author 2008; all rights reserved. Crowding-in effect?

Slide 24

Kinat Al Hilal s Prices Remained Stable Average Price (USD) per OC Cycle

Slide 25

MARKETS FOR HEALTH Take-aways Plausible attribution requires looking at multiple data sources and variables can be inexpensive by using secondary data National data sets allowed assessment of scale Post-endline review allowed assessment of sustainability Missed opportunity Potential crowding in could have been checked with interviews Systemic changes take time, but potential pay off is large Impact took over 10 years with extensive planning and negotiation

Slide 26

MARKETS FOR HEALTH Rigorous Attribution to Measure M4H Pilot in Nicaragua

Slide 27

Situation: 93% of informal sector workers lack health insurance Nicaraguan Social Security Institutes (INSS) goal: Extend health insurance programme to informal sector Increase health care utilization INSS contracted with 3 microfinance institutions (MFIs) to market voluntary insurance enroll beneficiaries collect premiums 17 predominantly private clinics to provide services Under capitated payment system Used an RCT to Measure impact of health insurance coverage on service utilization Remove selection bias (sick people more likely to sign up) Financing Function Health Insurance Pilot Managua, Nicaragua (20072008)

Slide 28

2007 baseline study conducted with 4,000 market vendors Vendors (sellers in market) randomized to receive Brochure about the health insurance program 6 months of free INSS insurance Nothing Subsidy winners randomized to register for insurance at INSS office or Local MFI 2008 endline study conducted with 2,608 of these vendors Two-stage least squares regression analysis conducted To measure effect of insurance on health service utilization and expenditures Controlling for endogeneity of insurance Study Design

Slide 29

Baseline Suggests Balanced Randomization ControlInfo Only6-Month INSS 6-Month MFI Age383738 Male.38.34.36.35 Years of Education9999 Married.78.71.69.68 # of Children2.22.02.22.0 Smokes.18.16.14.15 Income ($US)287255298263 Observations1111051659623 * Table refers to 2,608 respondents who participated in both baseline & endline

Slide 30

Main Findings Signing up Subsidized vendors more likely to sign up for insurance than controls 29% versus less than 1% Time & costs were main reason for not signing up Service utilization No net increase in healthcare utilization Total health expenditures for insurees decreased by 36% For those with chronic illness, health expenditures decreased by 89% Retention Less than 10% of enrollees were still paying for insurance a year later Those with subsidies were the least likely to be retained over time Main reasons for dis-enrolling Expense of premiums

Slide 31

MARKETS FOR HEALTH Take-aways RCTs allow rigorous attribution Remove selection bias and give accurate answers to important questions In general, systems approaches do not lend themselves to RCT evaluation due to scale & complexity RCTs are possible for some types of M4H pilots Pilot must allow randomization of beneficiaries into treatment and control groups Determine whether system intervention is worthy of scale-up Reporting failures makes important contribution to knowledge base

Slide 32

MARKETS FOR HEALTH OVERVIEW M&E FUNDAMENTALS M4H CONSIDERATIONS REAL WORLD EXAMPLES SUMMARY

Slide 33

MARKETS FOR HEALTH Session Summary M4H and other systems approaches pose important M&E considerations Need to explicitly monitor systemic change, scale, and sustainability Need to monitor after project ends to assess Expected J-curve impact Sustainability results Rely on plausible attribution most of the time Using mixed methods and triangulation Use rigorous attribution when it makes sense Pilot phase Testing an innovation Randomization of beneficiaries is possible (RCTS) Baseline equivalence possible (QEDs) Spill-over effects and contamination are unlikely Cost can be kept down