design, delivery and evaluation of eu rd policy - swg rrd · design, delivery and evaluation of eu...

Design, delivery and evaluation of EU RD policy

Luka JuvančičUniversity of Ljubljana, Biotechnical faculty

SWG RRD Seminar:

Accession to European Union in the Field of Agricultural and Rural Policies

Durres, 20-24 September 2010

The ‘forgotten middle’ of the EU RD policy

Policy Process

Changes & Challenges

EU - RDProgrammes

Objectives

Measures

Funding

ImpactDesign Delivery Evaluation

CMEF Assessment of Impacts

Source:

Rural development in pillar 2 RDPs

• A range of different interpretations at EU, national and local levels, influenced by policy context, history, institutions, interests

• This variety may influence the development of RD policies, thence their focus and their outcomes

• It may therefore be an important factor in the appropriate evaluation of impacts

Source:

RDP perspectives

Neo-mercantilist perspective: Pillar 2 is about supporting the farm (and agri-food) sector

• Axis 1 aids support continued occupation / viability of holdings, farm productivity, exports

• Agri-environment / LFA seen as critical for income, ‘broad & shallow’

• Axis 3-4 much lower emphasis

Neo-liberal view: P2 is about paying for public goods & adjusting to global markets

• Agri-environment seen as buying non-market goods, ‘narrow & deep’

• Axis 1 about adaptation and enhanced marketing/adding value

• Axis 3-4 aids business skills, start-ups, heritage & tourism,

strong diversification emphasis

- these views often link to positions on the future of CAP / P2

RDP perspectives (cont)

Agricultural: agricultural development is at the heart of / synonymous with rural development

Instrumental: a broader vision for RD is recognised, but P2 funding is explicitly targeted to support only that part which is closely linked to agriculture / forestry– typically those with relatively restricted P2 funds and/or

significant non-CAP rural funding

Broad-based: P2 is an important source of rural funding and significant RD needs / opportunities go beyond the farm and forestry sectors– those with depressed rural economies

– those with strong territorial governance

Source:

Design types: institutional maps

More

devolv

ed*

More stakeholder involvement

Type 1 (16 RDPs) – centralised, one lead MinistryBelgium:Wallonia Luxembourg

Greece Cyprus Italy:Puglia

Spain: Extremadura

Portugal Malta Spain:Catalunya

Slovakia Hungary

Romania Czech Republic.

Germany: Baden-Würrtemburg „ Mecklenburg-Vorpommern Sweden

Type 2 (12) – one lead ministry but designed by distinct axis: ‘sectoral’ stakeholder groups

England Wales

Latvia Lithuania Estonia

Scotland Finland

Bulgaria Slovenia

Netherlands France Austria

Type 3 (3) – divided, 2 lead ministries / agencies for different parts of RDP

Flanders (agency designed axes 3&4)

Ireland Denmark

Type 4 (2) – strongly devolved by territory = sub-programmes

Italy: Emilia-Romagna

Italy: Sicilia

Source:

The formal design processMany reported improvements, comparing with

EU-15 2000-2006 / NMS 2004-6 / SAPARD:

• more time available

• more thought given to role / relevance of RD funding

• more consideration (not always successful) to involve non-agricultural Ministries

• wider range of rural stakeholders included in consultations and/or working groups

• stakeholders’ expressed level of satisfaction is higher

• more developed integrated packages, co-operative working between partners, strategic focus

Source:

But…..Some stakeholders particularly influential

• Agri-lobbies often very dominant

• Smaller organisations may lack resources to engage

• Wider involvement highly variable

A degree of stakeholder pragmatism (even disillusion) about the potential of the RDP

– Limited synergy with other rural policy / programmes

– Restrictions from ongoing commitments

– Non-genuine / closed ‘consultation’ (narrowed over time)

– Lack of capacity to manage process effectively

– Limited funds mean other policies fill gaps

Source:

Policy Process


EU - RDProgrammes

Objectives

Measures

Funding



Source:

Only actors belonging to sectoral Administrations

Sectoral Administrations+ other actors:

Agencies/Provinces/Local municipalities/private

actors

From central/national to local level (multi-level governance)

Type of decision-making

Types of actors involved in decision making process

Dis

trib

ution o

f decis

ional pow

er Centralised at the top

level (Ministry/Central

Department/Payment

Agency)

Group a) Centralised and driven by

sectoral administration

Group c) Centralised and

multi-actors

Decentralised at lower

level (under the NUTS II

level)

Group b) Decentralised and

driven by sectoral administration

Group d) Decentralised and

multi-actors

TYPOLOGY OF DELIVERY

Source:

Table 4 - Typology of delivery and case studies

Only actors belonging to sectoral Administrations

Sectoral Administrations+ other actors:

Agencies/Provinces/Local municipalities/private

actors

Dis

trib

ution o

f decis

ional pow

er

Types of actors involved in decision making process

From central/national to local level (multi-level governance)

Bulgaria (Axes 1-2-3), Romania (Axes 1-2-3),

Hungary (Axes 1-2-3), Slovenia (Axes 1-2-3), Czech

Republic (Axes 1-2-3), Slovakia (Axes 1-2-3),

Lithuania (Axes 1-2-3), Latvia (Axes 1-2-3), Cyprus

(Axes 1-2-3); Denmark (Axes 1-2-3), Malta (Axes 1-2-

3), Portugal (Axes 1-2-3); Luxemburg (Axes 1-2-3);

France (national measures, i.e. 112); Wallonia (BE,

Axes 1-2-3); Flanders (BE, Axes 1-2), Wales (UK, Axis

1), Extremadura (ES, Axes 1-2-3), Cataluna (ES, Axes

1-2-3), Puglia (IT, Axes 1-2-3), Sicily (IT, Axes 1-2-3).

Greece (Prefectures); England (National

Agencies, Axis 2); Wales (UK, Axis 2, National

Agency); Austria (agri-environment, Regional

Chambers of Agriculture/Provinces/Municipalities)

Centralised at the top

level (Ministry/Central

Department/Payment

Agency)

Decentralised at lower

level (under the NUTS II

level)

Type of decision-making

France (214, Departments); Ireland (Axes 1 and 2,

Local offices, county-scale level), Meklemburg (D)

(Axes 1, 2 and 3), Baden-Wurttemberg (D) (Axes 1, 2

and 3).

England (UK, Axes 1-3-4, Regional Development

Agencies and Local partnerships); Wales (UK, Axis

3, Unitary Authorities); Emilia-Romagna (IT,

partially Axes 1-2-3, Provinces); Netherlands

(Axes 2 and 3, Provinces and agri-environmental

cooperatives); Flanders (Be, Axes 3-4, Provinces

and Municipalities)

Source:

Group a) Centralized and driven by sectoral administration

Selection

criteria

Application

form and

guidelines, calls

Assessment

and scoring of

applications

List of eligible

applications

Formal

approval of

applications

Axes 1 and 2

Danish Food Industry

Agency-Various Divisions

(Managing Authority)

DENMARK


Agency-Payment Division


Axes 1, 2 and 3




SLOVENIA

Ministry of Agriculture,

Forstry and Food (Managing

Authority)


Forstry and Food (Managing

Authority)

Agency for Agricultural Market

and Rural development

(Paying Agency)



(Paying Agency)

Axes 1 and 2 (individual

applications)




Regional Department of

Agricultural Policy


Monitoring Committee

Delivery

Phases


Agricultural Policy


PUGLIA (IT)




(Paying Agency)



Agricultural Policy



Agricultural Policy



Agency-Payment Division



Agricultural Policy


Group b) Decentralized and driven by sectoral administration

Selection

criteria

Application

form and

guidelines, calls

Assessment

and scoring of

applications

List of eligible

applications

Formal

approval of

applications

Local DAFF offices Local Ministry offices



Department of Agriculture,

Fisheries and Food (DAFF)



Environment and Consumer

Protection

Monitoring Committee Monitoring Committee

Department of Agriculture,

Fisheries and Food (DAFF)



Environment and Consumer

Protection

Delivery

Phases

IRELAND

MECKENBURG-

WESTERN

POMERANIA (D)

Axes 1 and 2 Axes 1, 2 and 3

Group c) Centralized and multi-actors

Selection criteria

Application form

and guidelines,

calls

Assessment and

scoring of

applications

List of eligible

applications

Formal approval of

applications

Ministry of Rural Development (Managing

Authority)

Delivery

Phases

GREECE

Farm Modernisation (measure 121)

Advisory Committee

of Management

Authority

Advisory Committe of

the competent

Region

Ministry of Rural

Development (Managing

Authority)

Regional Agricultural

Development

Divisions (Regions)

(Small Holdings) (Large holdings)

Ministry of Rural


Authority)

Prefectures, Regions and Local

Agricultural Development Centers


Ministry of Rural


Authority)

Group d) Decentralized and multi-actors

Selection

criteria

Application

form and

guidelines, calls

Assessment

and scoring of

applications

List of eligible

applications

Formal

approval of

applications

8 regional Developmet Agencies Sub-regional partnership

(operating at a county

level)



level)

Part of the programme

through:



level)

National Monitoring Committees

NGOs, Council for

Protection of Rural

England, Farmers,

Landowners and Tenant

Organisations

8 regional Developmet Agencies Business Link

Delivery

Phases

England National Policy Framework

Axes 1 and 3Regional

development policy

Department for Environment,

Food and Rural Affairs-DEFRA


8 regional Developmet

Agency

ERDF Structural

Funds

Programmmes plus

a range of businnes

and social support

services and

initiatives

Group d) Decentralized and multi-actors

Selection

criteria

Application

form and

guidelines, calls

Assessment

and scoring of

applications

List of eligible

applications

Formal

approval of

applications

Provincial Administrations or Regional offices of the Rural

Payment Agency (sub-contracted by Provinces)

Only for eligibility:

Rural Payment Agency

Selection based on policy

priorities:

Provincial Administrations

or regional offices of the

Rural Payment Agency

(sub-contracted by

provinces)

National Monitoring Committees

Provincial Administrations or Regional offices of the Rural

Payment Agency (sub-contracted by Provinces)

Delivery

Phases

Netherlands National Policy Framework

Axes 2 and 3 IBRA

Coordination Unit with the

participation of multiple

Ministries National Framework

Investment Budget

Rural Areas (IBRA).

It's a national

Programme

implemented

through

"performance

contracts" between

national and

provincial

administrations

Different times by types of measures

0

20

40

60

80

100

120

140

160

180

Farm income support measures Investment support in enterprises’

capital

Investment support outside

enterprises’ capital

Preparation of application Approval of simgle application Process payment for single application

Shorter implementation times

Special permissions and other

administrative fullfilments

Source:

Is LEADER delivery really more time-consuming ?

Not necessarily !!!!

Source:

Policy Process


EU - RDProgrammes

Objectives

Measures

Funding



Source:

Monitoring

Programming, reporting, monitoring, evaluation of EU RD policy: required procedures

STRATEGY REPORTING

•Community Strategic Guidelines

•National Strategy Plan

•Rural Development Programme

•Commission summary report

•National summary report

•Annual progress report

Quantified objectives/targets

Baseline indicators

In/output indicators

Result indicators

Impact indicatorsCMEF

Objective hierarchy

Impacts

Input Programme activities

Outputs

Results

Strategic

objectives

Specific

objectives

Operational

objectives

Objective

hierarchy

Definition Quantification

4 types of indicators (1)

• Input indicators:– measure and monitor progress in terms of the

annual commitment and payment of funds available for any operation, measure or sub-measure or priority within a programme (financial indicators)

• Output indicators:– measure the activity of the project/programme

and what the activities produces measured in physical or monetary units (number of projects, farms/firms, combined harvesters, km. roads, drinking water wells, micro enterprises etc.)

4 types of indicators (2)

• Results indicators:

– relate to the direct and immediate effect generated by the use of the project output.

– provide information on changes to the behaviour, capacity or performance of direct beneficiaries

• Impact indicators:

– refer to the consequences of the project and programme beyond the immediate effects on its direct beneficiaries (specific impacts) and the sector (global impacts)

Measure Output

(Operational

objectives, RDP)

Result

(Specific Objectives,

RDP)

Impact

(Overall

objectives,

examples)

Early

retirement

• 4,900 farmers enter

the scheme releasing

about 56,000 ha

UAA

• 23,652 small dairy

farmers leave the

market releasing

240,000 tons milk

quotas.

Average farm size

increases from 13.7 ha in

2002 to 15.9 ha by 2006

Share of farmers below

the age of 40 years is

increased from 14.4 % to

16.7 % in 2006

Number of dairy cows

increases from 2,2 cow to

3+ cows in 2006

Competitiveness

increased through

10 % reduction in

unit costs in 2006

Average farm

income raised by

10 % by 2006

Example of quantification

Expected outputs, results and impacts: definition of indicators and their quantification

(example 1)

Measure Output Result Impact

Less favoured areas

118,254 farmers with 942,000 ha entering the scheme

Differences in income levels reduced by 30 pct.

Environmentally friendly farming methods promoted

Rural population abstain from abandoning land

(Δ UAA =0 pct.)

Improvement of the environmental situation and preservation of biodiversity in RA

…..

Indicators

• An indicator is SMART:– Specific

– Measurable

– Achievable and agreed

– Relevant

– Time-related

• Must be defined at all levels: programme- measure – and project level

• Not only input and output indicators

Relation between indicators

Input Activity Output Result Impact

Funds ProjectCombined

harvesterHarvesting

capacityIncome

Evaluation

Theoretical backgrounds (1)

• Definition:

– To evaluate an intervention = to judge its value

On the basis of evidence

With regard to explicit criteria

• Added value of an evaluation:

– Knowledge and value judgement

– Learning and feedback

– Accountability

SocietyOrganisation

Theoretical backgrounds (2)• Feedback through evaluation (public sector):

Resources

Inputs

Evaluation

Services

Outputs

Impacts

Outcomes

Judging the value of outcomes

Inputs Outputs Outcomes

Audit Procedures & rules

Monitoring Management standards

Evaluation Citizents’ needs

Policy goals

Evaluation and the policy cycle

Impacts

Outputs

Results

Strategic

objectives

Specific

objectives

Operational

objectives

Needs

Problems

Stakes

Inputs

Resources

Evaluation and intervention logic

Society

Public intervention

Evaluation Ex-ante

Mid-term

Ex-post

Monitoring & Evaluation and Programme Implementation

Steps of evaluation process

1) Decision to evaluate

2) Terms of reference

3) Data collection, analysis

4) Report

5) Recommendations

6) Utilization

Decis

ion m

akers

Ste

ering C

om

mitte

e

Evalu

ation T

eam

The Terms of Reference

• Careful elaboration is important – good ToR helps tenderers design effective evaluation

• Prepared in good time

• A firm, but realistic time schedule

• Should include information on available data (baseline studies, monitoring system, reports etc.) – helps tenderer to assess needs for additional data

collection and design evaluation accordingly

• Selection criteria specified

• Budget limit included– Helps tenderer to establish realistic level of ambition in

evaluation design)

(Structure of the ToR)

1. Legal framework

2. Activity being evaluated

3. Uses of evaluation and why?

4. Main evaluation questions/issues

5. Methods (outline)

6. Available sources of information

7. Organisation of evaluation

8. Proposed timetable

9. Proposed budget

10. Required skills / backgrounds of Evaluation Team

11. Structure of the proposal

12. Attribution / selection criteria

Information which should be made available

• Data– Financial data– Monitoring data– Project/applicant database

• Documents– Programming documents– Reports (evaluations, baseline studies, project reports

etc.)– Applicant guidelines and information material– Case files– Internal guidelines on e.g. project selection criteria,

administrative procedures – …

Finding suitable data sourcesA typical evaluation includes several (or even all) of the following

techniques:1. Desk research (secondary data)

– Statistics, reports, surveys, studies

2. Personal interviews– With project management unit and other key actors eg. Authorities &

organisations (trade, labour, farmers) about progress/barriers, objectives, results)

3. Focus groups – More than just a ‘group interview’; interaction snd dialogue between

participants

4. Questionnaire surveys– Type of survey (live, telephone, postal) depending on target group

and required level of detail in information

5. Case studies of individual projects or beneficiaries– For qualitative insight, uncovering diversity, illustration of typical/best/worst

practices

Evaluation and EU Rural development policy: Common

Monitoring and Evaluation Framework (CMEF)

• Strategic monitoring: The RD regulation foresees strategic

monitoring of the Community and national strategies, link to

EU priorities

• Aggregation of outputs, results and impacts at the EU level

and help assess progress in achieving Community priorities.

• Need to better define baseline and indicators at the start of

the programming period to asses starting situation and form

basis for monitoring progress in delivering strategy.

• All requirements brought together in a single common

monitoring and evaluation framework (CMEF) to be agreed

with MS

CMEF: Evaluation

• Focus on ‘on-going evaluation’

• Key components: ex-ante; ongoing (annual progress report); mid-term and ex-post evaluations

• Use of independent evaluators

Reporting Requirements

Based on common monitoring and evaluation framework:

• Annual progress reports at programme level

• National summary reports on progress in implementation of the national strategy: 2010 – 2012 – 2014

• Commission summary report to the Council and the EP on the progress in implementation of the EU strategy and priorities: 2011 – 2013 – 2015

• Evaluation types: ex-ante (programme)ongoing (annual progress report)mid-term (2010)ex-post (2014)

Timing&delivery of EAFRD evaluations

2006 2007 2008 2009 2010 2011 2012 2013 2014

E x-post eva lua tion

2000 -06

Rural Development programmes

2007 -13Mid-

term

eva l.

RD

programmes

post 2013

P rog ramming post 2013E x-ante

eva l.

E x-post

eva lua tionE x-ante

eva l.

RD 2000-

2006

Ongoing eva lua tion