7_applying the synthetic control method (sard)
DESCRIPTION
Conference on Impact Evaluation: Methods, Practices, and LessonsAuditorium A, ADB Headquarters, Manila 11 July 2012TRANSCRIPT
![Page 1: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/1.jpg)
Illustrating with the Public Resource Management Program for Assam
Arnab Mukherji
Hiranya Mukhopadhyay
Conference on Impact Evaluation:
Methods, Practices, and Lessons
11 July 2012, ADB Auditorium Zone A
Applying the Synthetic Control Method
![Page 2: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/2.jpg)
Section 1
Evaluation and the Synthetic Control Method
2
![Page 3: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/3.jpg)
Evaluation is about Developing Parallel States of the world …
3
Before After - Restructuring Road
Implicitly: the missing counterfactual is … ?
![Page 4: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/4.jpg)
Public Policy Evaluations …
4
Have traditionally used the Before/After Design frequently. (e.g. See RBI’s State-Wise Analysis of Fiscal Performance)
Issues
– Easy to collect time-series data
– Identification rests on pre-intervention acting as a counterfactual for post-intervention
Concern:
– Does the pre-intervention status really give a good sense of what would have happened in the absence of the intervention?
![Page 5: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/5.jpg)
Before/After unable to disentangle other effects ….
5
FD/GSDP
Impact < 0
Observed Treatment Group
Counterfactual
FD/GSDP Program Works Program makes things
worse!
Impact > 0
Time (T) Time (T) T= Pre T= Post T= Pre T= Post
![Page 6: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/6.jpg)
Difference in Difference tells us …
6
Observed Treatment Group
Control Group
FD/GSDP
Time (T) T= Pre T= Post
ΔC
ΔT
Impact = ΔT -ΔC
Our best estimate of
what would have
happened in the absence
of the intervention
![Page 7: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/7.jpg)
A DiD Analysis raises …
7
A number of concerns centered on the validity of the control group:
Are the treatment and control groups similar on key dimensions?- Selection Bias
Or, do we use all untreated units in the control group symmetrically?
Strengths
– Clearly identified control group
– Omitted Variable Bias less of a concern
Needed micro data – until recently …
![Page 8: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/8.jpg)
Synthetic Control Group- Effect of Terrorism in Basque County
8
Discuss basic question – AER 2003 paper
Critical question 1-on which dimensions would we like synthetic Basque to be identical?
Critical question 2 -Which convex combination of states can be used to construct a statistically indistinguishable version of pre-intervention Basque?
– Solve for Variable weights
Control State weights
– Goal is reproduce pre-intervention outcome trends
Statistical balance on key predictors across Basque and Synthetic Basque
![Page 9: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/9.jpg)
Algorithm works as follows:
9
Let (YitI,Yit
N) denote an outcome in state i in year t; I, N indicate if the state received PRMA or not.
T0 is the intervention year thus, for all t in {1, …, T0-1} and for i in {1, …, N}, we want Yit
I = YitN.
Goal is identify a set of weight for each control state such that
is minimizing:
where is the set of predictors of Y for Assam and for other states and V is a diagonal matrix with specific weights for X.
![Page 10: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/10.jpg)
An Example
Suppose there are three states Assam (Program state), Maharashtra and Tamil Nadu
There are two predictors of the outcome variable (Y) : X1 and X2
The Minimization exercise is:
Min: V1(X1A- X1
MWM)2 + V2(X2A- X2
TWT)2
Subject to WT,WM>0 and WT + WM = 1
• But we still need to determine Vs
Vs are determined by minimizing the prediction error of the outcome variable (Y) for Assam from the synthetic Assam during the pre-intervention period.
![Page 11: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/11.jpg)
W vs. V
11
W gives us which states are to contribute to the synthetic control unit
V ensures that the chosen synthetic unit matches Assam in it it’s pre-intervention outcomes.
W is a function of V – we have a two stage simultaneous optimization problem – computationally intensive
Currently exists as a package in R – a statistical programming language.
![Page 12: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/12.jpg)
Main Advantages
It is a data-driven process- reduces selection bias.
Unlike Propensity Score Matching, does not require large data set.
Ideal for Macro interventions.
This method provides safeguard against extrapolation.
![Page 13: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/13.jpg)
Basque and Synthetic Basque
![Page 14: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/14.jpg)
Section 2:
Case Study: Public Resource Management in Assam
14
![Page 15: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/15.jpg)
PRMPA: Policy Response for Assam
15
![Page 16: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/16.jpg)
Focus on Two Key Components
16
Key Evaluation Challenge: What would have happened in the absence of the PRMPA?
![Page 17: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/17.jpg)
Section 3
Findings
17
![Page 18: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/18.jpg)
Two Outcomes
18
Own Tax Revenues/GSDP
Pre- Intervention Indicators
Synthetic Control –
Karnataka, Meghalaya, Tamil Nadu, Chhattisgarh, and Mizoram
Interest Payment to GSDP
Pre- Intervention Indicators
Synthetic Control –
Meghalaya, Uttar Pradesh, Orissa, West
Bengal
Variable Assam
Synthetic
Assam
Log(GSDP) 10.594 10.587
Non-Agriculture/GSDP 0.661 0.663
Per Capita # of Factories 0.057 0.059
Variable Assam
Synthetic
Assam
log(GSDP) 10.5937 10.60007
Debt/ GSDP 29.46% 31.10%
Average Effective Interest Rate 10.45% 10.22%
![Page 19: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/19.jpg)
Interest Payments/GSDP Ratio
19
![Page 20: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/20.jpg)
Own Revenues/GSDP Ratio
20
![Page 21: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/21.jpg)
Time Trends with Fiscal Deficit
21
![Page 22: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/22.jpg)
Broad Inferences
22
With the three years of follow-on data
Short-term gains have been achieved with respect to own tax to GSDP ratio - an increase in the order of 0.35% to 1.1% in the post-intervention period, averaging about 0.71%
No evidence of short-term gains achieved with respect to a decline in interest payments to GSDP ratio
Future trends will confirm – however, it would be interesting to learn about the pattern and nature of debt swap
Longer term horizon needed for full evaluation.
![Page 23: 7_Applying the Synthetic Control Method (SARD)](https://reader031.vdocument.in/reader031/viewer/2022020217/5450b3f1b1af9f11098b4dff/html5/thumbnails/23.jpg)
Thank You
23