regional policy theory-based evaluation: untapped potential for eu regional policy marielle riché...

Regional PolicyRegional Policy

Theory-Based Evaluation: Untapped potential for EU Regional Policy

Marielle RichéEvaluation Unit, DG Regional policy,

European Commission

Helsinki, 4th October 20124 October 2012


Content of paper

• Evaluations in EU regional policy: state of play• TBE are relevant for regional policy: throughout

policy cycle and for policy making• TBE still has much untapped potential: too much

focus on funds absorption and lack of clear theories of change

• 2014-2020: new opening for TBE (stronger focus on results, new regulatory requirements)

• Looking forward


Evaluations in EU regional policy

• Programme and evaluation management: MS or regions (EC: ex post evaluations)

• Programmes run for 9 years (2007-2013/15)• Cover research, ICT, urban generation, SMEs,

environment, social infrastructures etc. • May focus on specific territories, group of people,

sectors• May use "integrated approaches", may support

"innovation systems" – complex programmes


Evaluations in EU regional policy

• MS/Regions: some 266 evaluations since 2007 • Most on implementation issues and policy

relevance, few on outcomes (27%)• Most based on traditional qualitative techniques

and monitoring data• EC role: improve evaluation practice • Long-range aim: more rigorous evidence for policy

making, building shared knowledge ("Regio collaboration")


TBE are relevant for regional policy

• Programmes (should) rely on a theory of change• TBE are relevant throughout the programming

cycle (ex ante, in itinere, ex post)• Analyse the assumptions underlying the

programme • Assess the plausibility of programme

achievements as planned• Assess contribution, analyse mechanisms leading

to results, context, unintended effects


TBE are relevant for regional policy

• Fields of intervention where counterfactual impact evaluations not possible

• Complicated/complex programmes• Context analysis allows for policy adaptation and

replication• Communication on programme outcomes

(narratives)• Participative evaluations (partnership principle)


An untapped potential

• Programmes more focused on funds absorption than on results

• Evaluations carried out at the beginning of the period, focused on implementation

• No explicit theories in programmes• Over-confidence in indicators to measure actual

outcomes• "Myopia" of programmes (focus on first results)• Reluctance to commit to outcomes which are not

"under control"


2014-2020: a new opening for more TBE

• Political commitment for a "result-based" policy• New logical framework, taking into account

context and other factors, where impact is the contribution to observed results

• Stronger requirements for ex ante evaluations• Compulsory impact evaluations for each priority

axis of programmes (covers different interlinked interventions)

• Our preparatory work with Member States: guidance and pilots


How to implement these new requirements for 2014-2020?Our advice to Member States:•Use the evaluation ex ante to strengthen the programme theories of change•Be open to new approaches and allow for discussion on methods in Terms of Reference•Be realistic: take into account budget and time constraints, think of the primary use of evaluation (improve implementation or assess effectiveness? target audience?)


How to implement these new requirements for 2014-2020?Our advice to evaluators:•Help commissioners to refine evaluation questions•Propose and defend TBE designs mixing methods according to evaluation questions, timing, budget, planned use•Document data collection and analysis processes in evaluation reports


Conclusion

• We should not miss the opportunity given by ex ante evaluations

• Common goals: create a body of evidence on regional development

• Need for critical mass of good quality TBE: mix of methods framed by a rigorous design

• Need to develop strategies to minimise potential bias in data collection and analysis

• Need to develop criteria and standards for theory-based designs


Reference documents

On Inforegio Website:http://ec.europa.eu/regional_policy/information/evaluations/index_en.cfm

•Guidance documents•Impact evaluation center•Evaluation library (programmes)•EC evaluations

12RegionalPolicy

regional policy theory-based evaluation: untapped potential for eu regional policy marielle riché...

Documents