Counterfactual Impact Evaluation in the Framework of ESF

13/5/2015 RomeEvaluating impacts of hiring incentives in Italy

Jeannette MonierImpact Assessment and Evaluation Unit

ESF evaluations during 2007-2013

• By December 2013, the ESF Expert Evaluation Network had identified 721 evaluations (finalised and on-going)• Most evaluations are conducted at national or

multiregional level• Most were process evaluations• Counterfactual impact evaluations make up about

5% of the total number of evaluations identified

2

More impact evaluations required

• In this context, there is a need to supplement existing evaluation practice with approaches that generate much stronger evidence of the impacts of interventions: • One approach to capture impacts is through

counterfactual evaluations• Theory-based evaluation methods may be

required in addition to identify the underlying causal mechanisms

3

DG EMPL initiatives related to CIE

• Organisation of learning events on CIEs

• Publication of a Guide for ESF Managing Authorities based on contributions from Member States and evaluation experts

http://ec.europa.eu/social/BlobServlet?docId=10805&langId=en

• Two rounds of Pilot Projects to carry out ESF related CIEs

• Centre for Research on Impact Evaluation (CRIE)

4

CRIE

• CRIE work program consists of three blocks:

• Support to Member States

• Support to DG EMPL

• Dissemination activities

• http://crie.jrc.ec.europa.eu

5

Regulatory requirements for 2014-2020

• Results-orientation and thematic concentration make it necessary to design programmes that focus their resources on a few objectives in order to maximise their impact.

• Member States evaluations should help improve the quality of the design and implementation of programmes and also assess their effectiveness, efficiency and impact.

6

What impacts?

• Contribution to the objectives of each PA (Art. 56.3)

• Impact in relation to the EU2020 targets (Art.54)

• What does this mean?• Each SO should be covered• The extent will differ according to the duration

and size of the intervention, the nature of the result, available datasets, policy importance

• EU 2020: in most cases, qualitative analysis

7

ESF specific requirements

• Explicit requirement introduced for the MS to record and store microdata (on participants).

• Two evaluations of the YEI are required, one by end-2015, that will inform the overall process review of YEI and will feed into the mid-term budgetary review of the MFF, and one by end-2018.

• In order to capture impacts of YEI interventions, DG EMPL is investigating possible avenues to apply counterfactual methods in this area

8

Evaluation Plans

• Evaluation Plan submitted to MC no later than a year after adoption of OP: Art. 56,1 of CPR

• MC approves, reviews, amends: Art. 114, 110

• Follow up given to evaluation findings: Art.110

• From the Evaluation Plans received so far by DG EMPL, one conclusion to be drawn is that there is an increased number of counterfactual studies

9

Thank you for your attention!

Jeannette.Monier@ec.europa.eu

10