towards an enhanced monitoring and evaluation system: how m&e can become a tool for supporting...

Towards an enhanced monitoring and evaluation system: how M&E can become a tool for supporting programme management

Presentation for COP RBM premium group 6 April 2011

Benedict WautersDeputy director, Flemish ESF agency

Coordinator of the COP RBM

Where are the results?

“The most evident weaknesses which indicate the need for reform of cohesion policy are: …Methodological and operational problems that have prevented

both the appropriate use of indicators and targets – for which no comparable information is available - and a satisfactory analysis of “what works” in terms of policy impact.

A remarkable lack of political and policy debate on results in terms of the well-being of people, at both local and EU level, most of the attention being focused on financial absorption and irregularities.”

Independent report “An Agenda for a Reformed Cohesion Policy” delivered at the request of Commissioner for Regional Policy, Ms Hübner.

The discussion needs to go beyond (better) formulation of indicators!

We have to look at how M&E as a wholeSupports better programme management

Contents1. Theories of change as the basis of the M&E system2. Outcome indicator formulation is based on evidence and

relates to net effects3. Programme priorities should be based on a demand logic4. Ex ante evaluation: independent review of evidence5. Generate an evidence base for actions if there is none

available6. Programmes should be dynamic7. Independent evaluation authority8. Evaluation of TA = quality management9. Programme = policy dialogue10. Unique, complex, transformational projects should not be

impeded by the proposed enhanced M&E system

1. Theories of change as the basis of the M&E system

If Member States put forward measures/types of actions in their programmes, there should also be theories of change that explain how they intends to get from these actions to outcomes theories of change not the same as what is referred to

as “intervention logics”, although they should form a better basis for elaborating these

Within these theories of change, it should be clear where intrinsic needs of target groups are located (dimensions of well-being).

Theories of change should be supported by evidence

A simple cause-effect chain

Source: Adapted from Florida Department of Children and Families

Inside intervention: activity=CONTROL=OUTPUTS

External environment: change=INFLUENCE(varying degrees)

=OUTCOMES

Improvedhealth

Improvedcareer

prospects

A simple cause-effect chain




=OUTCOMES

Improvedhealth

Well-being

Improvedcareer

prospects

Direct influence

Contents

1. Theories of change as the basis of the M&E system2. Outcome indicator formulation is based on evidence and




2. Outcome indicator formulation is based on evidence and relates to net effects Based on the review of the evidence:

indicators can be formulated for outcomes reasonable targets can be set

Outcome targets should always be net, never gross Gross targets are totally irrelevant for programme managers

This is equivalent to companies being interested in knowing whether needs of market segments are fulfilled but not caring whether these needs are being fulfilled by their projects.

For outcomes that are very closely connected to the (elements of) financed actions within the theory of change and that become visible almost immediately after the action, attribution is not a (huge) concern:

Assumption that outcome is not due to something else happening at the same time can be supported (eg if before and after training you take test, the test result improvement probably due to the training –of course you can never be sure about this but for managerial purposes it is reasonable)

2 -continued

Deadweight) remains a concern (how much would have happened anyway

without financed action eg people would have been trained anyway) but this

is not relevant for short term managerial purposes (as Keynes said “in the

long run, we are all dead”) where the main issue is: do people learn what they

are supposed to learn in your training?

For outcomes that are further removed from the action, rigorous

impact evaluation is required

Managerial purpose shifts to the longer run

This may range from qualitative, in depth case studies to control

group style quantitative evaluations (or a combination)

These should therefore be planned for at the start of the OP

It is not necessary to cover all outcomes (see also principle 7)

Contents





3. Programme priorities should be based on a demand logic Within a priority outcome objectives should be

logically connected to each other within a comprehensive theory of change

The logic of a priority therefore should be demand and NOT supply led: eg. the priorities should not be formulated on the

basis of similar actions (eg. training) without taking into account the very distinct problems, needs and target groups these actions are supposed to address eg. training for employed middle managers is a different

issue than training for unemployed Roma. Putting these two on a same priority because it is all training reflects a supply led way of thinking.

Several actions

connected inone theory of

change = basis for a

priority

Source: E valuation of Flemish ESF OP 2007-14

Contents





4. Ex ante evaluation: independent review of evidence Ex ante evaluation at the programme negotiation

stage should consist, inter alia, of a serious evaluation of the evidence that a theory of change will work and will be measured correctly: For actions already proposed at the negotiation stage, the

evidence base that relates to the formulated theories of change for these actions, is reviewed.

If this is not given by programme management, it can be a task for the evaluators to find the existing evidence and link it to the programme

A judgment can then be given, based on this evidence base, on how likely it is that the theor(ies)y will be effective and that there are no conflicts with the policy context that will reduce this effectiveness.

Independence of the ex ante evaluation should be ensured.

Contents





5. Generate an evidence base for actions if there is none available For actions where the ex ante evaluation shows there is no

sufficient evidence base, small scale experiments to understand how these actions work and what their potential is, should be conducted. Small scale experiments can be carried out quickly and at reasonable

costs. They are not meant to be identical to an actual project but to focus on those elements of an action that are considered essential and to then assess the influence of this on intended outcomes (including intermediate ones).

This also feeds target setting as net effects obtained by the experiment provide a decent basis for extrapolation.

Actions where there is no sufficient evidence base should not be financed but put on hold or at least capped in terms of the amount of finances that can flow to them until a minimal evidence base can be generated.

Contents





6. Programmes should be dynamic For actions that are newly launched during

the life of the programme and that were not yet in the OP when it was negotiated, there should also be a theory of change that connects these new actions to the existing theory of change within one of the priorities. This allows to situate the new actions properly

within the various OP priorities. Ex ante evaluation and indicators formulation /

target setting should happen as for the actions already known at the beginning of the OP. This includes also the obligation to conduct small scale experiments if the evidence base is not sufficient.

Contents





7. Independent evaluation authority An independent evaluation authority judges whether

programme management acts responsibly throughout the programme execution period: The audit authority is the guardian of compliance with the rules.

However, there is yet no guardian that watches whether programme managers are also acting responsibly (ie. learning what works and what not/less and acting on this). Combined with weak evaluation cultures and capacities in various regions and countries this is serious threat to effectiveness.

A possible solution to both issues is to set up an independent evaluation authority (IEA), at the level of the NSRF or even the programme if the latter is large enough.

The programme managers still will have the initiative to determine, supported by the IEA , the plan for specific evaluations to be carried out. This plan should be based on the overview of where the knowledge gaps in the evidence base are greatest, or where the evidence base is becoming outdated. This means it is not necessary to cover all outcomes all the time.

7. continued The independence of these evaluations will be guaranteed by

the IEA. This IEA can be located together with the audit authority, as long as it is clear that it has another finality and requires other competences than those of auditors, or it could be established as a unit reporting directly to parliament or another position that guarantees some independence e.g. at the ministry of finance.

The IEA would have the critical mass to be a centre of evaluation expertise.

They should do some of the evaluation work themselves as well as act as an intermediary to reach consultants and academics in the market, providing for a sophisticated demand side, triggering capacity building and innovation in the market as well.

The IEA should be funded by OPs technical assistance to carry out evaluations for these OPs but they can also compete commercially to provide evaluation services for non-structural funds programmes and policies, thus building a dynamic of improvement and innovation in the evaluation market.

7. continued The IEA, similar to the audit authority, should annually provide

a judgment as to how “responsible” the programme manager has acted (did they learn about and adapt their programmes). If this assurance cannot be given the EC could block payments on the basis of this. The EC will however first engage in a dialogue between the IEA and MA.

The audit authority is in charge of assessing whether the IEA is itself in compliance with rules governing the work of the IEA.

The IEA would also be in charge of setting up and conducting the ex post evaluation. In order to do this, the IEA can draw upon the evaluations commissioned by the programme managers as well a complement these with extra evaluation activities, financed directly by the DG EMPL budget and with DG EMPL as commissioner.

It should also be able to conduct the necessary ex ante evaluations referred to earlier.

Contents





8. Evaluation of TA = quality management A clear distinction should be made between

evaluation of the technical assistance and evaluation of the actions/projects that have been financed. The former is part of quality management and

should be executed wholly by the Managing Authority itself. A variety of quality assurance tools exist ranging from ISO to EFQM to support this.

The latter is the remit of the IEA. Both are subject matter to be discussed during

annual reviews with the EC.

Contents





9. Programme = policy dialogue The debate between the Commission and programme

management should include participation of the relevant representatives of the government (incl. ministers) as many of the interventions are shaped by a particular policy context. If the policy context is a problem for effective ESF

interventions (as e.g. evidenced by evaluations), then this cannot be addressed at the level of programme management. This requires the presence also of the political level.

Annual reviews should be used to discuss policy level implications and the need to change certain policies. Actions that cannot be effective under current policy conditions may be blocked by the EC.

Contents





10. Unique, complex, transformational projects should not be impeded by the proposed enhanced M&E system These projects that are addressing many

different issues by containing many different, customised actions, often formulated in a bottom-up fashion (e.g. transformation of an entire company to increase quality of labour of employees) with non-standard outputs should not be impeded by the desire for a priori, generic, top-down, theories of change with standard ouptuts formulated at the programme level.

Output-oriented calls




=OUTCOMES

Improvedhealth

Well-being

Improvedcareer

prospects

Direct influence

Outcome-oriented calls




=OUTCOMES

Improvedhealth

Well-being

Improvedcareer

prospects

Direct influence

10 - continued It is necessary to address this with M&E in two

ways: At the project level, the need to be unique has to be

addressed by a sophisticated project appraisal process (e.g. using project cycle management approaches) that includes an ex ante assessment of:

a project level theory of change (eg using problem trees) with connected unique objectives, indicators and targets

as well as a screening of the evidence which is provided by projects applicants to substantiate that the project will be effective.

This is similar to what happens at the level of the programme between the EC and programme management

Next slides show example of system as used in Flanders

Outcome-oriented planning: from problems to objectives

Global objectives More disabled persons feel appreciated

Specific objective

More disabledpersons fulfill their potential

in a work environment

Global

Specific

Operational

TO TAKE ONNOT TO TAKE ON

Strategy 1: mobilise

supply

Strategy 2: increase

demand

Project intervention logic

MiEANS

Disabled retain

security when they take a

job

Disabled not afraid to look for

work

Disabled invited more

for job inteviews

Disabled find job more easily

Disabled feel appreciated

Physical environment more adapted to disabilities

Disabled have more self-

confidence

Adapting physical

environment is

expensive

Employers make betterassessment of potentialof disabled

Disabled areable to

performbetter

Training more

adapted to

disabledAdaptatio

n of training is expensive

Objective tree derived from problem tree

Non-disabledinteract moreeasily with disabled

GOAL

Operational objectives

Employers make better

assessment of potential

of disabled

Disabled areable to

performbetter

Disabledinteract more

easily with non-disabled

indicator

indicator

indicator

2. Indicators 3. Source

Project milestones

Budget/resources

Activities

Operational

Specific

GlobalWhat is

contributed toby project

Advantage for target

derived from using

what is available

What is made available:

knowledge, practice, attitudes, services...

What is donein the project

What needsto be the case, even if operational obj.

are realised?

What needsto be the case, even

if spec. obj. is realised?

1. Project description

4. Conditions

Outcome-oriented planning: summary of project

If target group does not believe advantage is worth acquiring, they will not use what is available!

What needsto be the case, even

if activities are executed?

Non-work environmentremains supportiveof disabled persons

1. x% more disabled are invited for job

interview each year 2. x% of disabled

obtain 80% score on standard competency

test3. x% of disabled score

min satisfactoriy in team role play

1. Diversity statistics2. Assessment center3. Assessment center

x% more disabled with mild physical

handicapshave fixed contract by

2010

CVS (client tracking system)

Share of excludedgroups in total FTEs of

organisations increased to 5% by

2020

Survey question tobe included in

regular sectoralsurvey

HR officers do notchange jobs

Disabilities of participants do not deteriorate beyond

being “mild”

-Number of new job openings remains constant

or grows-Anti-discriminationlaws continue to be

enforced-Disabled apply for jobs

Activities

Operational

Specific

Global

Social inclusionincreased

More disabled persons feel appreciated

More disabledpersons fulfill their

potentialin a work

environment1. Employers make betterassessment of potentialof disabled2. Disabled are able to perform better3. Disabled interact more easily with non-disabled

1. Project description

2. Indicators 3. Source 4. Conditions

Outcome-oriented planning: example summary of project

1.1 Info sessions for HR officers1.2 Adaptation of recruitment procedures1.3 Coaching for HR officers during recruitment...

Project milestones,budget/resources:

see planning

Outcome oriented planning: application form

Application form consists of list of standard questions: Analysis:

Target group description Process of involving stakeholders Problem tree Goal tree and strategic choices Lessons learnt from the past Policy context

Formulation of the project: Logical framework Planning (incorporating some elements of outcome mapping) Management (RACI table) Evaluation

Tools/guidance are provided to enable applicants to answer each question properly

Outcome-oriented planning: project appraisal Detailed appraisal grids (criteria) for readers of

project proposals linked directly to applicant

answers:

Relevance: the project should meet demonstrated and

high priority needs: 7 tables

Feasibility: the project should be well-designed and

deliver sustainable benefits to target groups : 5 tables

Good project management: 3 tables

Outcome-oriented planning : project appraisal Detailed appraisal grids (criteria) for readers of

project proposals linked directly to applicant

answers:

Relevance: the project should meet demonstrated and

high priority needs: 7 tables

Feasibility: the project should be well-designed and

deliver sustainable benefits to target groups : 5 tables

Good project management: 3 tables

Several proposal/appraisal questions

request that evidence be providedto reasonably assert that the project will

be effective

Appraisal is linked to the proposal questions

4 All components of an evaluation strategy are discussed and they are coherent. External stakeholders incl. the final target group will supply information in order to judge the project. At the same time it is clear who will execute the evaluation and who will follow-up and a budget is fixed.

3 All components of an evaluation strategy are discussed and they are coherent. External stakeholders incl. the final target group will supply information in order to judge the project.

2 All components of an evaluation strategy are discussed but they are not all coherent.

1 Only a limited amount of the components of an evaluation strategy are discussed.

0 Evaluation is confused with monitoring.

Proposal question on self-evaluation

•Goal of evaluation•Evaluation approach •Evaluation questions •Method of the evaluation•External or internal evaluators •Steering group evaluation: •Timing of evaluation and deliverables•Budget

Appraisal

Project applicants can score themselves in

advance!

Appraisal is linked to the proposal questions

4 All components of an evaluation strategy are discussed and they are coherent. External stakeholders incl. the final target group will supply information in order to judge the project. At the same time it is clear who will execute the evaluation and who will follow-up and a budget is fixed.

3 All components of an evaluation strategy are discussed and they are coherent. External stakeholders incl. the final target group will supply information in order to judge the project.

2 All components of an evaluation strategy are discussed but they are not all coherent.

1 Only a limited amount of the components of an evaluation strategy are discussed.

0 Evaluation is confused with monitoring.

Proposal question on self-evaluation

•Goal of evaluation•Evaluation approach •Evaluation questions •Method of the evaluation•External or internal evaluators •Steering group evaluation: •Timing of evaluation and deliverables•Budget

Appraisal

Project applicants can score themselves in

advance!

Self-evaluation stimulatedbut initially more intended to

generate evaluationculture at project level rather

than be useful directly for MA

(hard to aggregate)

10- continued

At the programme level, there can still be a more

generic theory of change, based on a broadly

formulated typology of actions that are eligible to

be financed within the complex projects.

This allows a programme level view of overall

effectiveness of projects that are in themselves

formulated in unique ways but that share an underlying

common logic.

The same principles as applicable to other types of

actions (see principles 1-9) can then apply here as well.

Towards a hybrid for outcome-oriented calls?

Top down intervention theory of change for a bottom-up call:

Could be used for M&E at programme level in parallel to project level bottom-up system?

10- continued This does mean that these projects will use two

reporting mechanisms: one at the programme level to feed the programme level

monitoring system. Of course, only paths in the programme level theory of change that are connected to types of actions present in the project need to be reported on.

one at project level to report on self-defined objectives and indicators that can be unique and more specific to the project than the programme level ones.

This system can thus marry – to some extent- a top down with a bottom up logic.

It is still important to understand that to really understand what is happening in and across these projects, more developmental M&E systems may be necessary at the level of the MA (e.g. SenseMaker can code and map 95,000 stories in 24 hours –see next slides)

Thank you!

towards an enhanced monitoring and evaluation system: how m&e can become a tool for supporting...

Documents

outcomestheories of

policy dialogueunique

policy debate

regional policy

better basis

reformed cohesion policy

reform of cohesion policy

terms of policy impact