VIII KONFERENCJA EWALUACYJNA

META-EVALUATION SOME REFLECTIONS VERONICA GAFFEY

Regional Policy

Meta-EvaluationSome Reflections

Evaluation Conference, Warsaw 12/13 November 2012

Veronica GaffeyHead of Evaluation

DG for Regional and Urban PolicyEuropean Commission

2RegionalPolicy

Regional Policy

3

Presentation1. Meta-Evaluation – just how ambitious do we want

to be?2. A more modest approach focused on the

accumulation of knowledge3. Appropriate level – EU/MS or both?4. The role of meta-analysis in ex ante evaluation 5. Elements in the new regulation to support

accumulation of knowledge, moving towards meta-analysis

Regional Policy

Regional Policy

4

1. Meta Evaluation• What is it?

Aim to improve decision making through systematic reviews on the effects of intervention

• Examples: Cochrane Collaboration – Health – since 1992 Campbell Collaboration – education, crime and

justice, social welfare – since 1999• Is our evidence base sufficiently strong?• Issues of internal & external validity

RegionalPolicy

Regional Policy

5

Our evidence base• Of 266 completed evaluations examined by

Expert Network in 2011, only 34% concerned effects; 66% implementation issues

• Pilot review process for DG REGIO finds few evaluations of Cohesion Policy of really good quality

• Difficult to judge quality of evaluations without terms of reference, budget and knowledge of context

RegionalPolicy

Regional Policy

6

2. Accumulation of Knowledge• Some experience here:

Meta-analysis of 200 evaluations across Europe prepared for Polish Presidency Conference on Evidence based Cohesion Policy

Forthcoming Regional Focus on What we have learned from counterfactual evaluations of innovation and enterprise support (Daniel Mouqué)

Internal Analysis on Transport and Environmental Infrastructure – likely to be transformed into guidance

• Insights rather than proofs; questions rather than prescriptions

Regional Policy

Regional Policy

7

3. Appropriate Level for Accumulating Knowledge

• Experience shows it can be done at EU level, if Terms of Reference and budget are public Context and method are explicit "Why it works?" is addressed as well as "Does it work?"

• Language issue can be overcome• DG REGIO starting to build a library of MS

evaluations• Also at Member State level, though

Difficult to generate critical massRegional Policy

Regional Policy

8

4. Meta-Analysis in Ex Ante Evaluation• Ex ante evaluation should be theory and

evidence based

• Testing the logic of intervention is at its heart – evidence from literature and other evaluations should be used

• Unpick the triangle in the next slide….Regional Policy

Regional Policy

9

Regional Policy

10

5. Elements in the New Regulation

• Clearer articulation of objectives with associated result indicators and baselines – success can be judged

• Requirement for an evaluation plan• Requirement for an impact evaluation of

each priority during the programming period• Summary report in 2021 – a meta-

evaluation?Regional Policy

Regional Policy

11

Possible Next Steps…• We continue to encourage more impact

evaluations – using different and combined methods

• Competition/Prize for good Terms of Reference?

• Structured exchanges of experience – based on themes

• Building communities of practice – involving public administrations & academic experts

Regional Policy

Regional Policy

Thank you for your attention

12RegionalPolicy