SECTION I.INDICATORS

Definitions, categories Indicators in the project cycle

Designing a system of indicators

References in IPA regulation

Art. 59. SMCInformation reporting obligation to IPA MC on

progress of […] indicatorsArt. 167. SMC for RD component

Examine results of implementation by reference to result indicators

Art. 169. SAR and SFRSectoral reports shall include result indicators

Definition

An indicator can be defined as the measurement of an objective to be met, a resource mobilised, an effect obtained, a gauge of quality ora context variable.

Definition

An indicator is made of:A definitionA valueA unity of measurement

Its measurement requires:A method (source of data, procedure of data

collection)Allocation of clear responsibilities to

personnel

General Criteria of Good indicators

OVI objectively verifiableSMART

SpecificMeasurableAchievable (acceptable, applicable,

appropriate, attainable or agreed upon)Relevant (reliable, realistic)Time-bound

General Criteria of Good indicators

CREAMClear Precise and unambiguousRelevant Appropriate to the subject at handEconomic Available at a reasonable costAdequate Provide a sufficient basis to assess

performanceMonitorable Amenable to independent

validation

General Criteria of Good Indicators

Another approach:Simple, clear and understandableUseful - # tiles in your office vs # education

sessions conductedValid – does it measure what it is intended to

measure and nothing elseSpecific – Should measure only the conditions or

event under observationReliable – should produce the same result when

used more than once to measure the same event

General Criteria of Good Indicators

Relevant – related to your workSensitive – will it measure changes over timeOperational –should be measurable or

quantifiable using definitions and standardsAffordable – should impose reasonable

measurement costsFeasible – should be able to be carried out

using the existing data collection system

General Criteria of GoodIndicators

Indicators should be expressed in terms of:QuantityQualityPopulationTime

For example, an indicator written for the program objective of “Increasing internet access” might specify:

“Increase from 20% to 60% (quantity) of internet access at the NUTS II regions (quality) of Turkey (population) by January 2014 (time).”

Categorisation of indicators

Challenging, since certain indicators can belong to several categories

Types of indicatorsRelated to the scope Context indicators: changes in the country

under evaluation, the location and the assistance provided

Programme indicators: resources, implementation, results, and impacts of an ongoing activity

Evaluation indicators: related to a programme's relevance, coherence, efficiency and effectiveness

Monitoring indicators: included in the monitoring system

Context indicators

Quantified information on the socio-economic and environmental situationcan express identified needs in quantitative termsExamples:

Economic development: GDP per capita, Direct foreign investment (% of GDP), Debt (% of GDP)

Quality of life and social well being: Under 5 mortality rate, Life expectancy, Primary education completion rate

Programme indicators

Relate to the effects of the intervention.Measure the extent to which the effects of a

programme are expected to change the socio-economic reality or the behaviour of socio-economic actors,

Expressing the quantified objective of the intervention

Examples:Number of start-ups, number of tourists visited the

beneficiary region, number of employment generated

Types of indicatorsRelated to their nature Elementary indicators: basic information from which more

complex indicators can be derived Derived indicators: ratio or rate, from the relationship between

two elementary indicators Compound indicators: combination of several indicators

(elementary or derived) Specific indicators: for an intervention, but not for comparisons Generic indicators: for comparing several activities Key indicators: for internal comparisons between different

activities of a programme and external comparisons with other programmes

Context indicators: for a country, population or a category of the

population

Types of indicatorsAnd more… Descriptive Management Policy Performance Qualitative Quantitative (specific number, percentage) Etc.

Quantitative indicators

Specific number: number, mean, or median but number of successful examination doesn’t

indicate the rate of success Percentage inidcates the rate of a

performance But rate of preformance doesn’t indicate the

size of the success Both to be used

Qualitative indicators

Imply qualitative assessment ‘Compliance with’, ‘quality of’, ‘extent of’ ‘level

of’ Changes in institutional processes, attitudes,

beliefs, motives, behaviors or perception of individuals

Qualitative data is more time consuming to collect, measure and assess

Hard to verify because involve subjective judgments

The logic we use in implementation

Area of influence external to Organisation

Area of control internal to Organisation

InputsResources

Output

Activities

OutcomeResults Impac

t

Level of indicatorsWhat longer term improvements are we

aiming at? (national goal) = impactWhat improvements are aimed at by the end

of the strategy period? = outcomes, resultsWhat strategic programs should be the focus

of the national response? = outputsWhat financial, human, material, and

technical resources are needed? = inputs

The logic from project aspectA public financial intervention – input

€ Millionsproduces some (physical) outputs, which are the

direct result of a certain operation,kilometres of a railroad constructed

beneficiary obtains some advantages, resultsreduced travelling time

intervention will affect not only final beneficiaries, but socio-economic environment - impactshigher GDP, increase quality of life

Input indicatorsMonitor the expenditure of the funds available

for any operation, measure or programme;Refer to the budget allocated

Financial indicators: progress in terms of commitment and payment in relation to its eligible cost

or to other technical or human resources elementsHuman resources indicator: number of working days

used in relation to total input planned

Output indicators

Looking forward to WORKMonitor the products of activitiesMeasured in physical units (e.g., Kms of

railroad constructed, number of firms supported, number of training days delivered etc.)

Result indicatorsLooking foreward to ACHIEVEMENTSMonitor the direct and immediate effects on direct

beneficiaries (target group)How will we know success or achievement when we see it? Are we moving toward achieving our desired outcomes?”Changesin behaviour, capacity or performance of

beneficiariesPhysical (reduction in journey times, number of successful

trainees, number of roads accidents, etc.)Financial (leverage of private sector resources, decrease in

transportation cost, etc.)

Result indicators

A core instrument for programme management

Impact indicators – difficult (impossible)Output indicators – information only about physical, not

socio-economic effects of an action special importance to result indicatorsFocus on design of high quality system of indicators

Sound analysis of the contextclear definition of the assumed causal chain, baseline, definition of the measurement method and a quantified target.

Impact indicators

Monitor the long-term effects beyond the immediate effects

Direct/Specific: consequences appear or last in the medium or long term for the direct beneficiaries.

Indirect/Global: related people or actors that are not direct beneficiaries

Require statistical data or surveys

Impact indicators

An instrument for strategy decisions(not a legal requirement)Decisive role in programming cycle: ex ante

quantification of impacts is an instrument for the strategic orientation of a programme during its planning phase;

only the impacts of a programme found ex post allow a final judgement on the success or failure

some impacts will only be measurable after a certain time of programme implementation, e.g. after 3 or 4 years.

Impact indicators

how to obtain their values?Data not necessarily obtainable from monitoring

system, only from evaluationValue might be due to factors external to the

programmeconsiderable time lag in availabilityBetter to limit to most important prioritiesNeed for a sound explanatory model to define

causal chain between programme input/output/results and impact values

An example of indicator-set for Entrepreneurship Development

Entrepreneurial Dynamismfocus: capacity to adapt to changing market

conditions creation of new enterprises: % of total no. of

enterprises EU15: 1% – 8%)enterprise survival rate after 3 years: total

number of start-ups in year ‘n’ created in ‘n-3’ against no. of start-ups in year “n-3” (EU15: 70% – 55%)

An example of indicator-set for Entrepreneurship Development

Regulatory and Business Constraints time involved in setting up a company; EU15:

1 – 24 weekscosts involved in setting up a company;

EU15: €100 - €2000business constraints: survey among

representative range of SMEs on lack of skilled labourforce; access to finance; technology change; infrastructure

An example of indicator-set for Entrepreneurship Development

Capital markets / financial conditions focus: availability of early stage venture

capital the total venture capital in % of GDP; EU15:

0,2-0-3% (USA: 0,4%)no. of Business Angel Network, no. of deals

initiated by BAN

An example of indicator-set for Entrepreneurship Development

Enterprise DynamicsInnovative Capacity

public expenditure on R&D as percentage of GDP

share of innovative SME in total (%): introduced new or improved products or processes

Knowledge based economynumber of SME using Internet for commercial

purposes

Use of indicators in the project cycle IPA (and even more SF) comprehensive set of objectives –

priorities, measure, etc – and wide range of actors – COM, ministries, OSs, OBs, beneficiaries;

OB broad set of information concerning the measure controls physical execution of projects physical output and financial indicators

OS less detailed information about specific measure and very little

information about individual projects objective of the programme priority result and impact indicators + some output indicators related

to OB’s efficiencyCOM

programme and priority level result and impact indicators

Challemge: select and record relevant data and direct them to relevant party

Indicators in programmingIntegration in programing;Establishment and Management in partnership

Involvement of suppliers and potential users of information project promoters, beneficiaries OBs – main suppliers, OS monitoring committees, European Commission, European Parliament and national parliaments, external evaluators, wider public, including civic organizations, official statistical services.

From temporary working group to monitoring platform

Indicators in programmingProportionality:

as complex as necessary and as small as possible impact and result indicators should cover priorities or

measures which represent the bulk of expenditure or are of strategic importance

Quality checksystem of indicators for coverage, balance, and

manageability;and individual indicators using quality criteria

relevance, sensitivity, availability, costs.

Indicators in programmingCoherence between programme documentsCoherence with indicators of established EC

policiesRole of ex ante evaluation

if benchmarks and past experience do not provide a sufficient basis for establishing and quantifying impact indicators

impact indicators complex task for programmersverify the causality between outputs, results and

impactsclose cooperation with planners

Indicators in programming

Programme Elaboration: Analytical part, definition of context indicators

Definition of Programme Strategy and Prioritiesdefinition of objectives at the Programme and

Priority level establishment output, result and impact and

core indicators

Indicators in programming

Planning Implementation Arrangementsdesigning the monitoring system: electronic

data processing, quality check of indicators,designing the evaluation system: planning

evaluation, with a description of indicator data needed to evaluate the Programme;

selecting indicators, information on which should be delivered by an evaluation exercise

Establishing rules and conditions for a smooth and efficient cooperation between monitoring and evaluation systems

Indicators in programming

Integration of Ex Ante Evaluation Ex ante evaluation as a parallel process to

Programme design: Close co-operation between the evaluators and programme designers as regards the indicator system, monitoring and evaluation arrangements

Examination of the evaluation recommendations and their possible consideration in the design of the Programme

Indicators during implementationData collection, updating and transferring

to users Task of OS and OBsConsolidation, improvement and rationalisation

of data Risk of excessive data requirementsOS to check periodically the reliability of the

information collected to provide additional guidance, if needed

Use and improvement of indicator system is a continuous task – strengthening administrative capacities

Indicators during implementation

Annual Reporting on ImplementationPreparation of the selected indicator data and

their preliminary interpretation for the Annual Reports

possible linkage between interim evaluation exercise and annual reporting

Indicators during implementationPresenting the data to the monitoring committee

(cont’d)Different knowledge and experiences of MC membersOS should:

Put quantitative information into its qualitative context,Reduce the volume of information provided, compared

to current experience,Present information in standardised manner,Undertake some preliminary analysis, highlighting

critical information, andUse appropriate presentation techniques

Turn monitoring findings into concrete actions

Indicators during interim evaluation

Evaluation of the programme performance as regards particular priorities or themes by using indicators as necessary

Review of indicators linked to a possible review of the programme strategy

Review of functioning of the monitoring system (quality of indicators, data collection and their transfer to the users)

Indicators during ex post evaluation

Indicators provided from Monitoring system (output and result)

Use of macro-economic models to evaluate impact

Indicators during evaluationIndicators are major source of information for evaluationsIndicators are most frequently used to measure

effectiveness and efficiency ratios

Objectives

Indicators Effectiveness

Efficiency

Measure/action

Financial/physical output

Actual/planned output

Outputcompared to cost

Priority Result (impact)

Actual/planned results

Result compared to cost

Programme

Impact (results)

Actual/planned impact

Impact compared to cost

Designing system of indicators

From input-driven implementation to a results-oriented indicator system

Element of judgement is required in addition to data processing

Designing system of indicators

A clear, focused strategyLimited number of prioritiesUnderstanding of intervention logic

Measures priorities programme Priorities should make explicit its underlying economic

and social rationaleExample

what is the mechanism through which capital grants are supposed to enhance the competitiveness of enterprises?

selection of appropriate indicators: instrument to clarify the content of measures and priorities

develop indicators within the discussion on the action

Designing system of indicatorsBaselinesBaseline data: initial value against which an indicator is

measured, e.g no. of SMEs in the regionConcepts:

Static: simple statement of a value for an indicator at a certain reference point in the past number of SMEs active in research in a certain year number of SMEs active in research supported by the

programme in a certain year of the pastDynamic: one projects the value of a certain indicator

during the programming period (baseline scenario or counterfactual situation)

Use depends on target area: region with a developed road network vs region without major roads

Designing system of indicatorsBaselines (cont’d) – information sourcesofficial statisticsProblematic

non-availability of data at an appropriate geographical level;

non-availability of data that is sufficiently disaggregated by sector;

delays in the publication of data; gaps in official statistics in relation to the

requirements of the programme (for example no distinction between full-time and part-time workers);

Designing system of indicators

Ex ante quantificationquantification of a target for an indicator is a

quality check of programminginstruments: use of historic time series and

the use of reference or benchmark valuesall outputs should be quantified at measure

levelin a next step quantification of result

indicators for the most important parts of a programme

Designing system of indicatorsCore indicatorsLarge number of indicators exist in a programmeCOM needs limited number of “core” indicatorsCore indicators specific for a programme

describe objectives of each priority in terms of expected results capture the core elements of the expected changes explaine in a qualitative manner in the programming document programme monitoring will pay particular attention to their attainment link with general policy frameworks, such as the Lisbon agenda

Common minimum core indicators programme indicators are not directly comparable across programmes physical and financial indicators used to make comparisons or

aggregations of data across similar programmes, priorities or measures

Horizontal issuesBC may wish to integrate horizontal objectives: sustainable

development, equal opportunities, crossborder cooperation, etc.

some general principles:should be embedded into the general indicator system of

a programme and not be separated into a specific indicator system.

The establishment of any indicator system is costly. Indicators for horizontal priorities should be applied first of all for measures that have a significant impact on a given horizontal theme.

step by step approach. It is equally important to be open to

where the nature of the assistance permits, the statistics shall be broken down by gender (SF)

Strategic environmental assessment: for significant effects on the environment of implementing a plan or programme;

Stages for setting up an indicator system

1. Pre-assessment of roles and responsibilities and capacity requirements

2. Defining outcomes based on agreement with stakeholders

3. Selecting Key Indicators4. Defining Baseline Data5. Selecting Results Targets M&E activities

The use of indicatorsMore indicators combined provide an image of

the situation, although often not a complete one. Other non objectively verifiable (measurable)

information is often eployed to improve understanding of the situation

Timely understanding the situation is needed by decision makers, in order to evaluate, assess, decide and apply decisions.

Decision making and application justify monitoring.

SECTION II.MONITORING VISITS AND

ON-THE-SPOT CHECKS Definitions and differences

Conducting Monitoring visits Risk assessment

On-the-spot-checklist

Monitoring visits & OTS checksOften confusing for both beneficiaries and project

managersMonitoring visitTool for data and information collectingEnabling managers:

to deal with problemsto find solutionsto adapt to changing circumstances in order to improve

project performanceto make personal contacts

Monitoring visit is not ... Auditing or inspecting Approving Beneficiary reports or contract amendments

… but also to help and support beneficiary

Monitoring visits & OTS checks

OTS checkpart of the controlcomplementary to desk control of filesto check the reality of actions and and the

reality of the expensescarried out by the OS/OB/CFCU

Monitoring visitsObjectivesInformation collecting on progress of project on the

spotActivities undertaken compared to ContractTiming of activities compared to milestonesFulfillment of indicators

Problem managementProblems already discovered by authoritiesProblems presented by ContractorRisk identificationDiscussing solutions to prevent or handle risksDefining scope of intervention

Monitoring visitsTypes1st monitoring visit

Usually during first 45 days of project implementation

Correctness of contract dataManagement and implementation capacity of the

ContractorDecide/update performance indicators of the projectIdentify the needs in the project implementationInform contractor on contractual obligations and

general EU rules Decide monitoring visit schedule for regular visits

Monitoring visitsTypesRegular monitoring visit

Correspondence between implemented, planned activities and the expenditures

Physical existence of purchased equipment Delivery and performance of sub-contracted

services Performance indicators The difficulties/bottlenecks Addendum/notification needs Early Warning and Irregularity Corrective actions (If needed)

Monitoring visits

TypesExceptional monitoring visit

In case of sudden problem to be solved urgently Personal contact is required (complex problem to be

clarified) Need to verify project data/results on the spot

(works)In case of risk identified with Contractor

Difficulties in communication Lack or regurarly delayed or bad quality reports

Monitoring visits

PlanningYearly monitoring visit schedule

Breakdown by months and contractors/beneficiaries

To be approved by OSRisk assessmentConsidering reporting period – before

deadlineDefining % of projects visited within a prioritySet-up of expert pool

Risk assessmentObjectives:

Optimal allocation of resourcesDefining samples to be checked on the spotDefining corrective/preventive measures

Risk is the expected value of the totality of damages incurring in a specific period of time

R = ptdt where pt: probability, dt: value of damage Task: defining probability and value of damage

Categorisation From highly probable to not probable From significant to unsignificant

Risk assessmentProgress (Dynamic) Risk Rate-variable

BudgetActivitiesProcurementVisibility & publicityNumber of objectionRules of originNumber of Addendum Rejected

Risk Rate by monitoring staffCompletion of project within budgetCompletion of project within planned duration

Overall Risk Rate

Risk assessment

Initial (Static) Risk Rate-constantNumber of activities,Total Budget,Experience in the relevant fields,Related EU project implementation experience,Is there external independent audit in the project,Number of partners,Legal status of the beneficiary,Co-finance ratio,Number of procurement above 5000€,Is there an accountant in the organization?

Risk assessment

Risk assessment criteria matrixRisk map or tableDefinition of monitoring visit needs

corresponding to risk categoriesResult: risk categorisation of projects

monitoring visit frequency defined

Monitoring visitsMonitoring expert poolNeed for technical knowledgeCapacity problems within authoritiesPublic procurement procedureManagement of expert pool

Planning of resource allocationVerifying performance

Time allocation Quality of performance

Confilct of interest, impartiality, confidentiality

Monitoring visitsPreparation Reminder of relevant information

Contractual obligations Timeschedule, milestones Outputs, indicators

Overview of relevant documents Call for proposal/tender documents, application/technical proposal Contract and annexes Notification on changes, contract modifications Progress reports Minutes of previous visits or meetings Documents of irregularity procedures

Define issues prior to meeting Preparing experts Letter of mandate

Conducting monitoring visitsNotification on visitFixing date with Contractor/beneficiary (informal)Notification letter (formal)

Date and place of visit Purpose of visit

Completion of activities Timeshcedul of implementation Actual status of technical progress Results achieved Achivement of indicators and visibility measures Moreover: assessment of problems, corrective/remedial actions

Participants from authorities Requested participants from Contractor/beneficiary Requested document (hint: office infrastructure to print minutes on the spot)

Conducting monitoring visitsConducting the visitPresenting letter of mandateDirected discussion based on questionnaire/checklist

Can the Contractor/beneficiary prove the status of the progress made?

Do activities undertaken aim at achieving contractual results?Are activities undertaken in line with EU and local

regulations/guidelines? Is achievment of indicators expected based on their actual

values? Have there been any risks emerged endangering project

completion? If yes, have appropriate corrective measures been undertaken?

Is there any need for immediate intervention? Is timing of progress appropriate, are there any significant

delays? Is there any need foe expert support to Contractor/beneficiary?

Conducting monitoring visitsConducting the visitMethods of information collecting

Checking, analysing documentsTargeted interviews with project managers,

project leadersInspecting of conrete outputs

Collecting/preparing documentary evidencesSupporting documents on transactionsPhotos on facilities equipped/built

Conducting monitoring visits

Conducting the visitPreparation of minutes

Project dataParticipantsLetter of mandate presented (signed by

Contractor)List of output inspectedList of supporting documentsForthcoming activities, expected milestonesFindings, remarks

Minutes of monitoring visits

Findings, remarks of Minutes (cont’d)Use of fundsProgress according to timescheduleAvailability of supporting documentsStatus of indicators, expected achievementsProject management, human capacityBest practice observedHorizontal aspects

Identification of problems encountered and measures undertaken/planned

Minutes of monitoring visitsExpert’s observations, recommendationsMeasures, actions to be undertaken by

Contractor/beneficiaryActionResponsible partyMeasurable result Deadline

SummaryCategories: further measure needed from Contractor

Y/N/further measure needed from authorities Y/NSupplementary documents needed Y/NRemarks (measures to be undertaken, follow-up visit to

be foreseen)Remarks of Contractor/beneficiaryTo be signed on the spot

Conducting monitoring visits

Follow-upSupplementary documentsMeasures to be undertaken by

Contractor/beneficiaryWatching deadlinesRequesting supporting documentsIrregularity procedureRegistration in MIS

Monitoring information daysSupplementary tool for monitoringGeneral information concerning project

implementationPresentation of End recepients guidelinesSpecific information on procedures of

Reporting Payments Contract modification Irregularities

Q&A session

OTS checksInternal control elementDetermine whether

the implementation is in accordance with the EU rules

internal control system is established and operating (grants)

Main areas coveredFinancial and accounting aspectsTechnical aspectsPublic procurementproject control

OTS checks

System checkto determine whether the system of internal

control is functioning efficientlyNational Fund at NIPAC and CAs

accuracy of expenditures presented to COMconformity of financial documents with the

originalsoverall objective to protect EU financial

intereststo prevent compensation obligation to the EC

for improper use of the EU funds.

OTS checks

System check by NFexistence of written procedures

Manual of Procedures conform to regulationsManual is actually used and systematically

updated all relevant documents on project

implementation existdocuments are kept in accordance with

archiving rules

OTS checks

System checkOS/OB/CFCU at Contractor/beneficiary

Accounting system is functionning properlyConformity of documents submitted with

originalsTo exclude double fundingTotality of tender documents compared to

submitted onesManagement structure

OTS system checks

Projects Financial Procedures & Systems Are there segregation of duties? What are the

arrangements to monitor and control project expenditure? Who prepares claims and who checks and signs claims?

Is the project meeting its spend targets? If not, why? Is a re-profile necessary (target spend -/+ 10%)

How are assets recorded and evidenced? Is the register up to date? (Record evidence seen)

Are adequate records being kept in relation to outputs/results/impacts? Provide details

OTS system checksFinancial managementDescription of the financial and administration

practicesHow do partners report their expenses?How are money transfers within the project

organised? VAT, Interests?Financial controlling practices

Follow- up of budget lines ;How the costs are followed? Which costs are reported under each budget line?How costs are controlled to stay within planned

budget

OTS project checksFinancial managementCheck of reported cost types

Staff costsOperational costsTravel and SubsistenceEquipments Indirect costs, overheads

Co-financingWhat are the sources?How are they recorded?

OTS project checks

Contract performanceEquipments, works in line with technical

specifications/designEquipments, works used for purposes and by

target groups and at the place defined in contract

OTS project checksContract performance – some aspects for training OTS

checkParticipants

number at the beginning and at the end compared to indicators

list of participants filled and how regularly during the training

target groups representedLength of the trainingCurriculum presented compared to originally agreedDistributed materials: quantity and qualityTraining tools used (entry/exit tests, exam,

certificate given)

Monitoring visit vs. OTS checksPotential over-lapping of scope

Contract performanceInspection of facilities

Different actorsMonitoring visits: mostly OB/OS task

managersOTS: mostly CA administrators/finance

officersWhy not harmonising the two? (HU CFCU

model)