balancing external representativity and ... - world...
TRANSCRIPT
ABCDE, World Bank 2016
{ noriega, pentland } @ mit.edu
Balancing External Representativity and Interference in Large-Scale Rural Experiments
Alejandro Noriega and Alex Pentland
MIT Media Laboratories MIT Institute for Data, Systems and Society (IDSS)
Field RCTs in Data-Rich Contexts
SAMPLE TARGET
POPULATION
Interference (SUTVA) Extrapolation
SAMPLE TARGET
POPULATION
Interference (SUTVA) Extrapolation
Field RCTs in Data-Rich Contexts
Overview
Spatial Dynamics of PROSPERA Context
PROSPERA – National Social Program
A Large Scale Experiment
Extrapolation and Target Population Experiment Design
Choice of Experimental Sample
Interference Gravity Model
Tradeoff Analysis
Interference and Representativity in Large Rural Experiments
Conclusion and Paths Forward
Algorithm and Results
National Social Program -- PROSPERA
• National-Wide system: – families below poverty line – Based on engagement and co-
responsibility
• 2nd Largest Conditional Cash Transfer program in the world – +8M families, +25M beneficiaries
• Services: – Health – Cash transfer – Nutrition – Productivity
Interference and Representativity in Large Rural Experiments
PROSPERA – Traditional communication means
Interference and Representativity in Large Rural Experiments
PROSPERA – Commutes
Interference and Representativity in Large Rural Experiments
PROSPERA – Commutes
Interference and Representativity in Large Rural Experiments
5,000 Approx. 5,000 women Jan 16
Interference and Representativity in Large Rural Experiments
Overview
Spatial Dynamics of PROSPERA Context
PROSPERA – National Social Program
A Large Scale Experiment
Extrapolation and Target Population Experiment Design
Choice of Experimental Sample
Interference Gravity Model
Tradeoff Analysis
Interference and Representativity in Large Rural Experiments
Conclusion and Paths Forward
Algorithm and Results
Interference and Representativity in Large Rural Experiments
PROSPERA – All Clinics
Interference and Representativity in Large Rural Experiments
PROSPERA – Eligible Clinics
Interference and Representativity in Large Rural Experiments
Extrapolation
PROSPERA – Eligible Clinics
Estimation error decomposition and non-random sampling
Interference and Representativity in Large Rural Experiments
Sampling Error
Treatment Assignment Error
Estimation error decomposition and non-random sampling
Interference and Representativity in Large Rural Experiments
Sampling Error
Treatment Assignment Error
non-random sampling
Interference and Representativity in Large Rural Experiments
Interference
Extrapolation
PROSPERA – Eligible Clinics
Overview
Spatial Dynamics of PROSPERA Context
PROSPERA – National Social Program
A Large Scale Experiment
Extrapolation and Target Population Experiment Design
Choice of Experimental Sample
Interference Gravity Model
Tradeoff Analysis
Interference and Representativity in Large Rural Experiments
Conclusion and Paths Forward
Algorithm and Results
Modeling Interference –- Road-and-Walk Route Distance
Interference and Representativity in Large Rural Experiments
𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑝𝑝𝑝𝑝𝑤𝑤𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑤𝑤𝑝𝑝
Gravity Model of Interference
𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝 𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑤𝑝𝑝𝑝𝑝𝑤𝑤𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑝𝑤𝑤𝑝𝑝
Interference and Representativity in Large Rural Experiments
Inter-clinic Interference Network
Interference and Representativity in Large Rural Experiments
Overview
Spatial Dynamics of PROSPERA Context
PROSPERA – National Social Program
A Large Scale Experiment
Extrapolation and Target Population Experiment Design
Choice of Experimental Sample
Interference Gravity Model
Tradeoff Analysis
Interference and Representativity in Large Rural Experiments
Conclusion and Paths Forward
Algorithm and Results
Sample search: Minimizing Interference
Greedy
Interference and Representativity in Large Rural Experiments
Sample search: Minimizing Interference
Algorithm 1
Interference and Representativity in Large Rural Experiments
Interference Minimization
Interference and Representativity in Large Rural Experiments
Interference vs. Representativity Tradeoff
Interference and Representativity in Large Rural Experiments
Interference vs. Representativity Tradeoff
Interference and Representativity in Large Rural Experiments
Representativity – KS Covariate Distance (Kolmogorov-Smirnov)
Interference and Representativity in Large Rural Experiments
Target Population
Sample
KS Distance For relevant covariates :
• Social lag • Illiteracy • Education • Indigenism • Newborn Weight • Diseases • Distance to clinic • Clinic infrastructure
Interference vs. Representativity Tradeoff
Interference and Representativity in Large Rural Experiments
Interference and Representativity in Large Rural Experiments
Bi-Objective Sample Search @k=25
Interference (SUTVA) Representativity (KS)
Interference vs. Representativity: Bi-Objective Sample Search
Interference and Representativity in Large Rural Experiments
• Evaluate direct and ‘spillover’ health effects (experiment is on!)
• Evaluate models of interference: gravity model, road network vs. spatial, communications (CDRs)
• Bi-objective sample search: optimization bounds, better bi-objective heuristics.
Research Paths
Forward
Interference and Representativity in Large Rural Experiments
Modeling Interference (SUTVA violations)
Interference and Representativity in Large Rural Experiments
Interference vs. Representativity
Bi-Objective Sample Search (Algorithm 1)
Interference and Representativity in Large Rural Experiments
Innovating the Mexican social health and financial system
Interference vs. Representativity
Bi-Objective Genetic Algorithm
Innovating the Mexican social health and financial system
Interference Minimization
Modeling Interference –- Road Network + Walk
Distance
Innovating the Mexican social health and financial system
Health Intervention
Potential Projects with Mexico – National and Mexico City
Geospatial Analysis: • Macro spatial dynamics of health clinics, rural & urban
populations of beneficiaries, and mobile phone coverage
Potential Projects with Mexico – National and Mexico City
Geospatial Analysis: • Macro spatial dynamics of health clinics, rural & urban
populations of beneficiaries, and mobile phone coverage
Potential Projects with Mexico – National and Mexico City
The collaboration
Potential Projects with Mexico – National and Mexico City
The collaboration
Free SMS and CDRs
RapidPro plattform