1 the need for control: learning what esf achieves robert walker
Post on 31-Dec-2015
214 Views
Preview:
TRANSCRIPT
1
The Need for Control: Learning what ESF achieves
Robert Walker
Identifying impact
Time
Outcome
Control orcounterfactual
Policy success?
Introduction of policy
Impact
Defining the counterfactual
Time
Outcome
Introduction of policy
Control group
Action group
(Treatment group)
Defining control groups
• Random assignment
• Random encouragement designs
• Matching areas
• Individual matching
• Propensity score matching
• Difference in difference
• Regression discontinuity designs
Defining the control groupRandom assignment
12
6
9
53 4
10
12
87
11
Action group
Control group
Random number =
Defining the control groupRandom assignment
1
26
9
5
3
4
10
8
7
11
Action group
Control group
12
Random = 7 = ControlNumber Group
Advantages of random assignment
• Action group and controls same (with large numbers) with respect to:– Observed characteristics– Unobserved characteristics
• Unbiased estimate of mean impact• Possible to state degree of confidence that
estimate is a true measure of impact
Non- participants
Eligibleindividuals
General population
Participants
Controlgroup
Programme group
Random assignment
Determineeligibility
Ineligibleindividuals
Random assignment
Adapted from Orr (1999)
Non- participants
Eligibleindividuals
Jobseekers Finland
Participants
Controlgroup
Jobsearch training
Random assignment
Determineeligibility
Ineligibleindividuals
Jobsearch training - Finland
Malmberg-Heimonen and Vuori (2005)
If staff thought would benefit &Did not express a preference for training
677
19 employment offices
338
430 247
20.3% 19.5% find employment
Seek informed consent 18% asked by post, 28% asked face to face agreed
Jobcentre Plus office
New Deal25+
New Deal forLone Parents
Working Tax Credit
ERADEmployment Retention and Advancement
Demonstration
P C
RA
Mandatory WFI
Voluntary JC+ visit
P C
RA
P C
RA
JB+ initiated contact
Participants
Targetpopulation
Non-participants
Encourage to participate
No encourage-ment
Random assignment
Participants Non-participants
Randomised encouragement design
IMPACT
Defining the control group
Geographic areas(Limited equivalence)Population
Area matchingUK - Educational Maintenance Allowance
Propensity Score Matching
Programme
Control
Criteria for inclusion in control
Have the combination of characteristics
making them likely to be included in the programme group
PSM Norway, Rehab and
Activation
Rønsen and Skarðhamar (2009)
Months to finding work
Participants
0 5 10 15 20
Controls
(Social security recipients)
Months until 25% find
work
Controls 20
Participants 11
Difference in difference design O
utc
om
e
TimeT1 T2
Programme group
Control group
CounterfactualDifference at T2 (D2)
Difference at T1 (D1)
Difference at T1 (D1)
IMPACT(D2 - D1)
Regression discontinuity design
Measure of need/eligibility (at time T1)
Outcome (at time T2)
(Measure of need)
High
High
Low
Low
Regression discontinuity design
Measure of need/eligibility (at time T1)
Outcome (at time T2)
(Measure of need)
High
High
Low
Low
Threshold
Regression discontinuity design
Measure of need/eligibility (at time T1)
Outcome (at time T2)
(Measure of need)
High
High
Low
Low
Threshold
Non-participantsParticipants
IMPACT
Regression discontinuity design: Extended UB entitlement, Austria
IMPACT
14.8 weeks extra unemployment
Unemploymentduration
Age
Lalive (2008)
Characteristic Random assignmen
t
Random encouragement
Matching RegressionDiscontinuity
Individuallevel
Areacontrol
s
Propensity score
matching
Characteristics of policy intervention (1)Measure systemic effects
No(Yes, for Cluster
RA)
No No Yes No Yes
Multiple target groups
Possible for a few
Difficult but possible for a
few
Possible for a few
Not easy
Yes Yes
Multiple policy components
Possible for a few
Difficult but possible for a
few
Not easy
Competing policy designs
Possible for a few
Difficult but possible for a
few
Not easy
Characteristic Random assignmnt
Random encouragement
Matching RegressionDiscontinuity
Individuallevel
Areacontrol
s
Propensity score
matching
Characteristics of policy intervention (2)
Need to measure second order effects
No(or only by using quasi experimental methods)
Yes(but only by further supplementary individual level matching, i.e. further quasi-experimental
methods)
Targeted on hard to reach group
Difficult Not easy Difficult Not easy
Difficult Not easy
Targeted at areas
No (Yes, for Cluster RA)
Not easy Yes Yes Yes
Characteristc Random assignme
nt
Random encouragemnt
Matching RegressionDiscontin-ty
Individuallevel
Areacontrol
Propnsty score
matching
Ease of implementationDisruption of existing policy
Yes No Yes No No No
Requirement for informed consent
Yes No Yes No No No
Ethical concerns
Real for all prospective evaluation since people are placed at risk of harm due to possible negative effects of previously untested policy for the
common good
Scale: sample sizes
Small Small Very large
Mod-erate
Large Moderate
Data requirements
Low Low Moderate Mod-erate
High Moderate
Characteristic Random assignmnt
Random encouragement
Matching RegressionDiscontinuit
yIndividuallevel
Areacontrol
s
Propensity score
matching
Quality of impact estimatesUnbiased impactEstimate
Yes Yes No No No No(yes at margin)
Contamintion of control group
Moderate Moderate Moderate Low Low Low
Control for unobservable variables
Yes Yes No No No No
Overall control Excellent Excellent Poor Poor Moderate Moderate
External validity
Limited Limited Moderate Limited Higher Higher
Conclusion• No evaluation method is perfect• Evaluation without a counterfactual is very, very
likely to wrong and misleading• Only random assignment (and regression
discontinuity designs) produce unbiased estimates
• It is impossible to be confident that controls are adequate
• Control groups are generally better than no control group – they force evaluators to think about the characteristics of a good counterfactual
top related