ExPlorATIvE STuDy oF ProgrAMME/ProjECT CoMPlETIoN rEPorTS

october 2007

from danida’s evaluation department - development cooperation, october 2007

1

Explorative Study of Programme/Project Completion Reports

Development Associates October 2007 Toldbodgade 19B, 2. Sal DK-1253 Copenhagen K Denmark

Published by: Ministry of Foreign Affairs of Denmark Evaluation Department Asiatisk Plads 2 1448 Copenhagen K Denmark E-mail: eval@um.dk The publication can be downloaded from: www.evaluation.dk ISBN: 978-87-7667-803-6 (print version) ISBN: 978-87-7667-804-3 (internet version) Responsibility for the content and presentations of findings and recommendations rests with the author. The views and opinions expressed in the report do not necessarily correspond to the views of the Ministry of Foreign Affairs of Denmark.

TABLE OF CONTENTS Page Abbreviations 3 Executive Summary 4 1 Introduction 6 2 Method of work 6 3 Statistical Overview of PCRs and Responses Given 7 4 Lessons Learned 2006 23 5 PCR Formats 24 6 Harmonisation and Alignment Efforts and Danida Requirement for PCRs 26 7 Wider Knowledge Sharing of Lessons Learned 27 8 Annual Update of Lessons Learned and PCR Statistics 29 Annexes: Annex 1: Terms of Reference Annex 2: Lessons Learned Annex 3: Detailed Proposal for Adjustments to PCR Guideline and Formats

LIST OF ABBREVIATIONS AIDS Acquired Immuno-Deficiency Syndrome ASP Agriculture Support Programme BFT Technical Advisory Services CBoH Central Board of Health CD Compact Disc CHD Centre for Humanitarian Dialogue DAC Development Assistance Committee Danida Danish International Development Assistance DKK Danish Kroner EMG Evaluation Management Group ENRECA Enhancement of Research Capacity EVAL Evaluation Department FDA Focal Development Area FFU Forskningsfagligt Udvalg FSPS Fisheries Sector Programme Support GHA Ghana Highway Authority HIV Human Immuno-deficiency Virus HSPS Health Sector Programme Support ICRC International Committee of the Red Cross MFA Ministry of Foreign Affairs MLD Ministry of Local Development NGO Non Government Organisation PCR Programme Completion Report PDB Project Data Base PhD Doctor of Philosophy PMA Plan for Modernisation of Agriculture RUF Rådet for Udviklingsforskning SESD Skills and Education Sector Development SEAQIP Seafood Export and Quality Improvement Programme ToR Terms of Reference TSPS Transport Sector Programme Support

EXECUTIVE SUMMARY Objective of study 1. The purposes of the Explorative Study of

Programme/Project Completion Reports (PCR) are to contribute to the establishment of a simple system of (scheduled/annual) electronically based and user friendly synthesising of lessons learned from PCRs, and to contribute to the development of a systematic and on-going knowledge-sharing of Danish lessons learned among Ministry of Foreign Affairs (MFA)-staff, partners in programme countries and non-programme countries, and for the wider public, accessible via the internet and/or the intranet.

Self assessment 2. PCRs are designed to present an

assessment of programmes and projects in order to provide documentation on the use of Danish funds and to generate lessons learned of general interest to future development aid activities. In contrast with evaluations undertaken by independent consultants, preparation of PCRs is a self assessment by the project management and the highest project authority.

Statistics 3. During 2006, a total of 315 PCRs were

received by the Evaluation Department. Of these 108 used one of the three new formats for PCRs introduced in January 2006. Apart from NGO projects (a total of 39) all programmes and projects should use the new format. These PCRs together account for some DKK 2.8 billion of approved grant budget. About 80 % of PCRs are for projects below DKK 5 million, with around 13 % being between DKK 5 and 30 million, and the remaining 7 % being above DKK 30 million.

4. Looking at all programmes and projects, irrespective of their type, the highest portion (about 40 %) of programmes and projects are located on the African continent. Asia and South East Asia together account for about 35 % of the total PCRs. The PCRs cover a total of 51 different countries.

5. Most of the PCRs cover activities

implemented within the last five years. Programmes and projects that have been implemented faster than planned or within the planned period account for about 55 % of the projects.

6. The new formats for PCRs have

introduced rating of performance. Irrespective of the criteria and project size, and taking all scores (a,b,c and d) into account the analysis shows that 95 % of the ratings in PCRs are either an “a” or “b”. This shows that according to the programme and project managers there has been a general high degree of performance.

7. The fact that there is such a little variance

in the scores given, i.e. that 60 % of all scores given is a “b” and that 35 % of the scores is an “a” raises the question how valuable the scoring system is. While it certainly has a value to ask the people who are involved in the programmes and projects about their opinion of the interventions, caution should also be taken in making too strong conclusions from these results. In particular because it is a “self assessment”, but also because the cultural settings in which the projects are implemented in a number of cases makes it difficult to “officially” make a negative judgement.

Lessons learned 8. The report presents a condensed version

of the lessons learned as expressed in the PCRs received during 2006.

Formats in use 9. In general it is the impression that the new

formats are an improvement compared to older formats.

10. It is recommended that:

• The NGO projects should use the same formats (either format 1 or 3) for project completion reporting as all other bilateral projects, which are required to submit PCRs.

• The PCR guideline be edited in order to provide additional clarity to the users which have to fill these in.

Time of completion of projects 11. In the PDB it is unclear when a project is

completed. 12. It is recommended that:

• The PDB is amended so that there are two fields to be marked for completion of PCRs: One field for marking that the draft PCR has been uploaded and one field for marking that the final accounts have been approved and the project therefore can be regarded as also being financially completed.

13. It should also be considered to upload the

final accounts and audit report in the PDB in order to have all reports for a programme/project in one location.

Wider use of PCRs 14. At present the Evaluation Department

makes use of the PCRs. However, it is found that there is scope for a wider use of PCRs among other MFA staff as well as among partners and for the general public.

It is recommended that:

• PCRs are presented in a systematised manner on the web site of the Evaluation Department for reasons of transparency;

• The condensed version of the lessons learned and statistics of PCRs are published on the web site of the Evaluation Department;

• The PDB is improved to include a user-friendly search function related to PCRs.

Method for updating lessons learned and PCR statistics 15. It is recommended that:

• An annual update is made of the lessons learned as expressed in PCRs and of the statistics that can be generated from PCRs.

16. In order to secure a smooth preparation of

the annual update and in the best possible quality, the methodology to be used in this process can be divided into two main parts: • A quality assurance procedure being

applied through the year, each time a PCR is received, and

• The process of preparing lessons learned and statistics on the basis of quality assured PCRs

7

1. INTRODUCTION This report is prepared in accordance with the Terms of Reference (ToR) for the “Explorative Study of Programme/Project Completion Reports (PCR)”. The overall purposes of the study are to contribute to the establishment of a simple system of (scheduled/annual) electronically based and user friendly synthesising of lessons learned from PCRs, and to contribute to the development of a systematic and on-going knowledge-sharing of Danish lessons learned among Ministry of Foreign Affairs (MFA)-staff, partners in programme countries and non-programme countries, and for the wider public, accessible via the internet and/or the intranet. The recommendations of the study should also be seen in the light of adding an additional instrument to the ones used by Danida in its project cycle management, including linking with the ongoing monitoring of the support programmes. While the PCRs have a function in documenting lessons learned the study also discusses how it can be used in preparation of new programmes and projects. Further, the explorative study should also be seen as a response to questions raised by the Auditor General. The main study was carried out during the period 14th of November 2006 to 5th of February 2007. Comments received were incorporated during the last part of April 2007. The Consultant1 would like to express his sincere thanks to all officials and individuals met for the kind support and valuable information, which the Consultant received during his work and which highly facilitated the work of the Consultant. This report, however, contains the views of the Consultant, which do not necessarily correspond to the views of Danida. The report addresses the findings and recommendations under the below mentioned headings: Method of Work; Statistical Overview of PCRs and Responses Given; Lessons Learned 2006; PCR Formats; Harmonisation and Alignment Efforts and Danida Requirement for PCRs; Wider Knowledge Sharing of Lessons Learned, and Annual Update of Lessons Learned and PCR Statistics.

The ToR for the study are attached as Annex 1 and a condensed version of the lessons learned as expressed in PCRs during 2006 is shown in Annex 2. Annex 3 presents a detailed proposal for adjustments to the PCR guideline and formats. The report including its annexes is also delivered on a CD Rom. The CD Rom in addition contains the Excel Sheets developed as part of the study, and an electronic version of all PCRs received during 2006. 2. METHOD OF WORK The study has been carried out as a desk study supplemented with discussions and interviews with MFA staff. The Consultant has received a presentation of the main source of information, the Project Data Base (PDB), in which the PCRs are stored. A list of projects for which PCRs have been received during 2006 was generated by the PDB office in an Excel spreadsheet and served as an important reference

1 Mr. Bjarne Larsen, External Consultant to Danida.

point throughout the study. The Consultant received an electronic version (either in Word or in PDF format) of the major parts of the PCRs from the Evaluation Department. Following receipt of the PCRs the Consultant read all these. Simultaneously data from the PCRs were entered into an Excel spreadsheet, which had been developed by the Consultant on the basis of the list of projects generated by the PDB office. The type of data to be entered were selected from the point of view that it should subsequently be possible to address the scope of work of the ToR for the explorative study, including generating relevant statistics. Subsequent to the reading of PCRs a condensed version of the lessons learned, as expressed in the PCRs received during 2006, was prepared. Following completion of data entry the statistical module of the Excel spreadsheet was developed. Based on the statistics generated the Consultant prepared a draft paper summarizing the findings from the PCRs as an input to the Evaluation Department which had to contribute to the Annual Performance Report 2006 within this area. The statistics also served the purpose of preparing the full statistical overview of PCRs and responses given. Included in the explorative study were also discussions on how best to establish a wider knowledge sharing of lessons learned, the conduction of a comparison between the list of PCRs received during 2006 and the Programme and Project Orientation (PPO) as included in the Annual Reports of Danida, as well as an assessment of the strength and weaknesses of the old and new PCR formats. 3. STATISTICAL OVERVIEW OF PCRs AND RESPONSES GIVEN PCRs are designed to present an assessment of programmes and projects in order to provide documentation on the use of Danish funds and to generate lessons learned of general interest to future development aid activities. In contrast with evaluations undertaken by independent consultants, preparation of PCRs is a project management function and the Programme/Project Manager usually makes the reports. While the authors of PCRs may in many instances be rating projects in which they have been directly involved, an endorsement of the PCRs is also to be made. In the new format for PCRs, the Chairperson of the Programme and Component Steering Committees as well as the Head of the Danish Representation must endorse PCRs for programmes and components. The Head of the Danish Representation in the country must endorse projects below DKK 5 million or, in case there is no Danish Representation, the endorsement must be given by the relevant Head of Department. 3.1 Programme and Project Size During 2006, the Evaluation Department was informed about the uploading of a total of 315 PCRs to the PDB. Of these 108 used one of the three new formats for PCRs2. Apart from NGO projects (a total of 39 projects) all programmes and projects should use the new format. Five completed projects mentioned in the PPO 2005 were yet to submit PCRs. The received PCRs together account for some DKK 2.8 billion of approved grant budget. Of this amount DKK 2.5 billion has been spent by the programmes and projects covered by the PCRs received during 2006. To avoid double counting it has in the following been taken into account that budgets for approved programmes

2 For a discussion of the PCR formats, see section 5. NGO projects have their own format. A total of 39 NGO projects submitted a PCR during 2006.

include component budgets. Therefore approved component budgets of about DKK 700 million have been excluded from the calculations3. About 80 % of PCRs are for projects below DKK 5 million, with around 13 % being between DKK 5 and 30 million, and the remaining 7 % being programmes above DKK 30 million. Table 1 Number of Programmes and Project PCRs According to Size Of the DKK 2.8 billion approved grant budgets, the largest programmes and projects account for 76 % of the total budget. The smallest projects (below DKK 5 million) account for only 10 %. Table 2 Distribution of Total Programme and Project Budget According to Size, for

which PCRs were Received in 2006.

3 A total of about DKK 670 million actual spending by components have been excluded from the calculations for the same reason.

Distribution of total programme and

budget according to size10%

76%

14%

Projects below DKK 5 million Programmes above DKK 30 millionProjects above 5 and below DKK 30 million

Number of PCRs

252

21

42

Projects below DKK 5 million Programmes above DKK 30 millionProjects above 5 and below DKK 30 million

3.2 Distribution of Programmes and Projects According to Grant Type. By distributing programmes and projects according to the type of grant it is seen that in terms of number of projects the two largest groups are research projects and local grant authority projects. Table 3 Number of Programmes and Projects According to Grant Type.

However, in terms of total approved grant budget the absolute largest category is programmes and projects above DKK 5 million4, which account for DKK 2.1 billion. Research and NGO projects each account for DKK 210 million.

4 It should be noted that of the total 63 programmes and projects (as per table 1) above DKK 5 million, 30 programmes and projects are allocated to the categories research projects (6), humanitarian projects (9) and NGO projects (15). The remaining 33 programmes and projects are shown in table 3.

Table 4 Approved Grant Budget According to Grant Type

3.3 Geographical Distribution of Programmes and Projects for which PCRs were Received

in 2006 A characteristic feature of the programmes and projects is that the absolute majority (about 77 %) is implemented in a specific country5. About 8 % are regional projects, i.e. implemented in more that one specified country. The remaining 15 % are termed international, i.e. they do not have a specific country coverage. Table 5 Type of Project: Country, Regional or International. Number of PCRs. Looking at all programmes and projects, irrespective of their type, shows that the highest portion (about 40 %) of programmes and projects are located on the African continent. Asia and South East 5 It should be mentioned that any project which is implemented within a given country is included, whether it is e.g. a programme component, a project which is the responsibility of the Danish Representation, a project implemented by an international NGO or a research project carried out through a cooperation between an international university and a local partner.

Type of Project

243

26

46

Country Regional International

Asia are the two other major geographical areas where the programmes and projects are located. These two areas together account for about 35 % of the total PCRs. The PCRs cover a total of 51 different countries. Table 6 Distribution of PCRs According to Region. Number of PCRs. Looking specifically at the 15 Programme Countries6, at total of 158 PCRs were received during 2006. Tanzania, Vietnam and Nepal accounted in total for almost 60 % of these PCRs. The high number of PCRs from activities in Tanzania can mainly be explained by a backlog of projects (80 % of the PCRs) and that Tanzania is a large programme cooperation country. The high number of PCRs from activities in Vietnam is a combination of Vietnam being a large programme cooperation country, that the first phase of the Fisheries Sector Programme Support closed as well as a certain backlog of projects (about 1/3 of the PCRs). In Nepal there was a closure of many projects during 2005 and 2006, which in part can be explained by the general political and security situation in the country. It might also be a reflection of the mode of cooperation where it has been necessary to support especially good governance and human rights through a high number of smaller interventions. About ¼ of the PCRs in Nepal were for start-up projects financed under the Private Sector Development7 Programme.

6 As per 1st of January 2006. 7 The Business-to Business Programme replaced the Private Sector Development Programme as of June 2006.

Regional Distribution of PCRs - number

127

15 7

43

67

0 6 4

46

020406080

100120140

Africa

South

Ame

rica

Centr

al Am

erica

South

Eas

t Asia As

ia

Pacif

ic Re

gion

Europ

e

Midd

le Ea

st

Intern

ation

al

Table 7 Distribution of PCRs According to Programme Country. Number of PCRs. As far as the Programme Countries are concerned, more than DKK 2.2 billion8 worth of programmes and projects submitted PCRs during 2006. Of these more than 90 % of the budgets were from 7 countries. This can mainly be explained by the fact that in each one of these countries at least one major programme or project was completed9. In Bangladesh the “Water Supply and Sanitation Sector Programme, Phase I”, in Ghana the “Transport Sector Programme, Phase I”, in Vietnam the “Fisheries Sector Programme, Phase I”, in Bolivia the “Agricultural Sector Programme, Phase I”, in Zambia the “Health Sector Programme, Phase III” and in Mozambique the “Emergency Rehabilitation of the Flood and Cyclone Damaged Electrical Infrastructure” project. In Kenya the “Agricultural Support Project, Phase II” as well as the “Community Based Nutrition Programme” submitted PCRs in 2006.

8 For PCRs covering programmes and projects in the programme countries, an amount of about DKK 2 billion was spent. 9 This does not imply that the support to the sector is not continued. In several cases a new phase of the programme support is now in the process of implementation.

Distribution of PCRs according to Programme Country

11

15

106

1

84 5

26

1

42

9

25

4

05

1015202530354045

Bang

lades

hBe

nin

Bhuta

n

Bolivi

a

Burki

na Fa

soEg

ypt

Ghan

aKe

nya

Moza

mbiqu

eNe

pal

Nicara

gua

Tanz

ania

Ugan

da

Vietna

m

Zamb

ia

Table 8 Distribution of Programme and Project Budgets According to Programme Country. DKK Million. For PCRs received during 2006.

3.4 Sector Distribution of Programmes, Components and Projects The PCRs have been distributed according to “sectors” using the DAC codes. The largest group of projects according to this distribution is research projects10, which has not been assigned to a specific sector. The second largest group of projects are within the good governance sector, covering a wide range of interventions in support of democracy, legal development, media, decentralisation and human rights. The social sectors (education, health, population and other social infrastructure) together also features prominently with 53 projects. Agriculture as a sector and “emergency assistance and reconstruction” are two other major groups of projects.

10 Old (2002) DAC code 16381, equivalent to the new (2006) DAC code 43082.

Distribution of approved programme and project budgets in DKK Million according to Programme Country, for PCRs

received during 2006

559

1 7

230

4 2

450

164 13781

0,156 66

330

189

0

100

200

300

400

500

600

Bang

lades

hBe

nin

Bhuta

n

Bolivi

a

Burki

na Fa

soEg

ypt

Ghan

aKe

nya

Moza

mbiqu

eNe

pal

Nicara

gua

Tanz

ania

Ugan

da

Vietna

m

Zamb

ia

Table 9 Sector Distribution of PCRs. Number of PCRs. In terms of budgets, the largest sectors are agriculture, water, transport, education and health. In all cases this is due to the completion of major sector programmes, which budget wise account for a large proportion of the total approved budget of programmes and projects covered by the PCRs received during 2006. Comparing with the number of PCRs according to sector, it is seen that on average research projects are small. Considering that one major Good Governance programme in Nepal is included, it is also a feature of good governance interventions that each of these typically are small.

Sector Distribution of PCRs - Number

2013

513

48

5

15

4 1

16

28

2 28

4 7

82

2

26

95

0

10

20

30

40

50

60

70

80

90

Edu

catio

n

Hea

lthP

opul

atio

nW

ater

Gov

/Civ

il so

ciet

yC

onfli

ct, p

eace

and

sec

urity

Oth

er s

ocia

l inf

rast

ruct

ure

Tran

spor

tE

nerg

y

Bus

ines

sA

gric

ultu

reIn

dust

ry

Trad

eE

nviro

nmen

t

WID

Rur

al D

evel

opm

ent

Res

earc

h

Food

Aid

Em

erge

ncy

Ass

ista

nce

Trad

e U

nion

Oth

ers

Table 10 Budget Distribution According to Sector. DKK Million. For PCRs received in 2006. 3.5 Project Start, Completion and Time Management Most of the PCRs cover activities implemented within the last five to six years. More than 80 % of the projects for which PCRs were received in 2006 were planned to start during the new century. With a typical planned duration of 5 years for the new generation of sector programmes and components, an explanation for the significant number of PCRs with a start year from 1999 to 2001 could be the expansion of the development aid budget and preparation of new sector programmes during these years, combined with a certain delay in actual stating date and “no cost” extension of programmes and components. However, a detailed analysis only to a certain extent confirms this: For 1999 as a starting year, 7 of the 30 PCRs were for new programmes and components. For 2000, the figures were 11 out of 40 PCRs; and for 2001 only 1 out of 50 PCRs. The analysis shows that it is rather a reflection of a higher number of smaller projects approved during the period 1999 to 2001. The lower number of PCRs for projects with a planned starting year 2002 and 2003 can partly be a reflection of an overall lower development aid budget. The increase from 2003 to 2005 can partly be explained by approval of a higher number of smaller projects with a short duration.

Budget distribution according to sector - for PCRs received in 2006

228286

6

547

12221 23

414

7013

696

19 1 34 7 4864

2121

46 50

100200300400500600700800

Educ

ation

Healt

h

Popu

lation

Water

Gov/C

ivil s

ociet

y

Confl

ict, p

eace

and s

ecuri

ty

Othe

r soc

ial in

frastr

uctur

e

Trans

port

Energ

y

Busin

ess

Agric

ulture

Indus

tryTra

de

Envir

onme

ntWI

D

Rural

Dev

elopm

ent

Rese

arch

Food

Aid

Emerg

ency

Ass

istan

ce

Trade

Unio

n

Othe

rs

Table 11 PCRs Received in 2006 Distributed According to Planned Start Year for Programmes and Projects. Number of PCRs.

As far as the planned project start is concerned, the above explanations for receipt of PCRs during 2006 should be supplemented by the fact that there has been a certain backlog of projects, which ended before 2005. A total of 144 PCRs (about 46 %) received during 2006 had a completion date before 1st of January 2005. Of the 124 projects that ended during 2005, 35 were completed during the first half of 2005. In principle projects completed during the first 6 months of a calendar year should submit a PCR before the end of the year. The overall picture is therefore that of the 315 PCRs received during 2006, 136 were received in time and 179 were received after the deadline of 6 months. Table 12 Actual Project Completion Year. Number of PCRs. To assess the degree to which programmes and projects have been implemented within the planned duration, an analysis of the PCRs has been carried out to determine to which degree projects have been on time, have been implemented faster than planned or have been overrun. On time has been defined as a project that has been implemented within the exact planned number of months. Underrun has been defined as a project, which has been implemented at least 1 month faster than

Project Start Year

2 04

1 39 7

30

40

50

35 35

44

51

4

0

10

20

30

40

50

60

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

Project Completion Year

1 1 1 314

22 17

42 43

124

47

0

20

40

60

80

100

120

140

1995 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006

the planned duration. Overrun has been defined as a project, which has been implemented within a period of time that has been at least 1 month longer than planned. Programmes and projects that have been implemented faster than planned or within the planned period account for about 55 % of the projects. Table 13 Programmes and Projects on Time, Underrun and Overrun. Number of PCRs. An analysis of the PCRs for the 144 projects which have been overrun reveal that about half of these projects are for activities in 5 countries: Tanzania (25), Vietnam (14), Nepal (10), Bolivia (9) and Bangladesh (8). Taking into account that the total number of PCRs from each of these countries varies, the largest proportion of projects overrun are found in Bolivia (9 out of 10) and Bangladesh (8 out of 11). A sector wise analysis of the projects that have been overrun shows that the highest number of projects are from the good governance sector (28), research (20), agriculture (17), education (14) and business (12). Considering that there is a different number of projects within each of these sectors, the proportion of the total number of projects that have been overrun within each of the sectors have been calculated: The sectors with the largest proportion of overrun projects are the business sector (12 out of 16) followed by education (14 out of 20) and agriculture (17 out 28). In terms of approved budgets for projects for which PCRs have been received in 2006, the proportion of projects that have been overrun is 54 %. This percentage is partly explained by the fact that 5 sector programmes which in terms of budgets were heavy, were extended in time: The “Water Sector Programme” in Bangladesh, the “Fisheries Sector Programme” in Vietnam, the “Community Based Nutrition Programme” in Kenya, the “Agricultural Sector Programme” in Kenya and the “Support to Education and Skills Development Programme” in South Africa.

Time Management - Number of projects

19

152144

0

20

40

60

80

100

120

140

160

Projects underrun Projects on time Projects overrun

3.6 Ratings of Programme and Project Performance in PCRs The basis for the ratings of programme and project performance is the score (a, b, c or d)11 according to a number of set criteria in the three new PCR formats. In some cases a score of performance has also been made in the old formats. Where it has been possible to translate such a score unambiguously into the score used in the new format, this has been done. An example is that the use of “very satisfactory” in an old format has been translated into an “a” in the new format for a given criteria e.g. “fulfilment of project development objective”. Therefore a PCR using an “other” format will in some cases contain a score according to the new PCR format. In one case format 1 was used for components of a programme. In this case the score of “fulfilment of objective” and “fulfilment of the poverty alleviation objectives” were transferred to a format 2 and the remaining (for crosscutting criteria) format 1 scores omitted. Therefore the number of scores according to format 2 criteria shown in the analysis is higher than the number of PCRs prepared in format 2 by the respective programmes and components (as shown in Chapter 5). Due to these adjustments made, the number of scores in format 1 and 2 are not directly comparable with the information provided in table 18 (see chapter 5) on formats used. Irrespective of the criteria and project size12, and taking all scores (a,b,c and d) into account the analysis shows that 95 % of the ratings are either an “a” or “b”. This shows that according to the programme and project managers there has been a general high degree of performance. Table 14 Distribution of All Scores Irrespective of Criteria and Project Size. Number of

Scores. The following shows the scores according to the criteria used in PCRs for programme level interventions (format 1), i.e. fulfilment of development objectives, poverty alleviation, gender equality, environment, good governance and HIV/AIDS response. It is a general feature that there are only “a” and “b” scores for fulfilment of development objectives and poverty alleviation, while “c” only exist for the cross cutting issues. There are no significant differences in the ratings among the four cross cutting issues, except that environment received a high proportion of very satisfactory scores. The programmes covered are the “Water Sector” in Bangladesh, the “Fisheries Sector” in Vietnam, the “Education Sector” in South Africa and the “Health Sector” in Zambia. In addition the regional 11 An “a” represent very satisfactory, “b” satisfactory, “c” less satisfactory and “d” unsatisfactory. 12 I.e. including all scores from format 1, 2 and 3 together.

Total number of scores

63

110

10 0

0 20 40 60 80

100 120

a b c d

project “Man, plants and environment in Western Sahel” covering Burkina Faso and Senegal is included. The same is a grant to the “ENRECA-collaboration between Institute for Development Studies, University of Nairobi, and Centre for Development Studies, Copenhagen”, covering the period 1992 to 2004. The difference in total number of scores among the criteria is due to the fact that some criteria have been mentioned as being non-applicable. Table 15 Scores on Fulfilment of Criteria for Programme Performance. Number of

Scores.

Fulfilment of development objective

1

5

0 00 1 2 3 4 5 6

a b c d

Poverty Reduction

2

3

0 00

1

2

3

4

a b c d

Gender Equality

2 2

1

00

1

2

3

a b c d

Environment

4

1 1

00

1

2

3

4

5

a b c d

Good Governance

2 2

1

00

1

2

3

a b c d

HIV/AIDS Response

2

1 1

00

1

2

3

a b c d

The following shows the scores according to the criteria used in PCRs for component level interventions (format 2), i.e. fulfilment of overall component objectives and poverty alleviation. Performance has been high according to both criteria, except in one case (for a component of the “Education and Skills Development Programme” in South Africa) for the poverty alleviation objective. Included are components under the “Water Sector Programme” in Bangladesh, the “Fisheries Sector Programme” in Vietnam and the “Education and Skills Development Programme” in South Africa. Table 16 Scores on Fulfilment of Criteria for Component Performance. Number of

Scores. The following shows the scores according to the criteria used in PCRs for small project level interventions (format 3), i.e. fulfilment of project objectives. Again, the absolute majority of scores indicates a high degree of performance. Only 5 have rated the projects less satisfactory according to this criterion (projects in Bolivia (2), Burma, Sri Lanka and Tanzania). It should be mentioned that in some cases the PCRs filled in in format 3 have had two or three objectives mentioned. Scores for these additional objectives have not been included in the calculations. An additional 23 small project objectives were rated “a”, 37 rated “b” and 2 rated “c”. Table 17 Scores on Fulfilment of Criteria for Small Project Performance. Number of

PCRs. A distribution of scores according to major sectors has been carried out for projects below DKK 5 million. The major sectors are the social13, water, good governance14 and the productive15 sectors. It 13 Covering education, health, population and social infrastructure.

Overall Component Objective

2

10

0 0 0 2 4 6 8

10 12

a b c d

Poverty Reduction

3

10

1 0

0

2

4

6

8

10

12

a b c d

Project Objectives

45

76

5 0

0 10 20 30 40 50 60 70 80

a b c d

is seen that only the social and good governance sectors have “c” ratings. The productive sector has the highest performance in terms of the largest proportion of “a” scores. Table 18 Scores on fulfilment of project objective according to sector. Number of scores

for small projects. The fact that there is such a little variance in the scores given, i.e. that 60 % of all scores given is a “b” and that 35 % of the scores is an “a” raises the question how valuable the scoring system is. While it certainly has a value to ask the people who are involved in the programmes and projects about their opinion of the interventions, caution should also be taken in making too strong conclusions from these results. In particular because it is a “self assessment”, but also because the cultural settings in which the projects are implemented in a number of cases makes it difficult to “officially” make a negative judgement. It should also be considered that it is the degree of contribution to the fulfilment of the development objective for the programme (format 1), the fulfilment of the intermediate objective of the programme (format 2) and the project objectives (assumed also to include the development objective) (format 3) that are assessed. It is generally known that these are difficult to assess, also

14 Covering government/civil society, conflict prevention, peace and security. 15 Covering business, agriculture, industry and trade.

Distribution of scores according to sector - social sector

small projects

7

18

2 0

0 2 4 6 8

10 12 14 16 18 20

a b c d

Distribution of scores - Water sector small projects

1

5

0 00

1

2

3

4

5

6

a b c d

Distribution of scores - Good governance small projects

12

29

3 0

0 5

10 15 20 25 30 35

a b c d

Distribution of scores - Productive sector small projects

1415

0 00

2

4

6

8

10

12

14

16

a b c d

because the donor only contribute to reaching these objectives, which will only be accomplished (long) after the completion of the intervention. The definition of the scores “a” to “d” might also be able to explain a part of the very significant number of high scores: An “a” is defined in the PCR guideline as “No need to adjust plans and strategy” and “b” as “Minor problems may arise and small adjustments may be necessary”. This definition, as applied in the sector assessments, which deals with an ongoing programme, might not be clear in terms of assessing a completed programme or project. A more clear definition relating to the degree of fulfilment of objectives for an “a” might be “Objective fully achieved, very few or no shortcomings” and for “b” it might be “objective largely achieved, despite a few shortcomings”. A “c” might be defined as “Objective partially achieved” and a “d” as “very limited achievement, extensive shortcomings”. For sector programmes an assessment is also made each time there is a sector review, using the same scoring system. On a spot check basis, and looking at the most recent sector assessment, a comparison has been made between the sector assessments and the scoring in the PCRs for a few sector programmes: The “Water Sector Programme” in Bangladesh, the “Fisheries Sector Programme” in Vietnam and the” Health Sector Programme” in Zambia. The comparison shows that there is no systematic difference between the rating in the PCR and the sector assessment of the same criteria. i.e. some of the ratings are identical and some are slightly better in either the PCR or in the sector assessment. Only in one case exists a significant difference – a “d” is given for the HIV/AIDS response in the sector assessment while an “a” is given in the PCR. 3.7 Correspondence Between PPO and List of PCRs A specific issue is to which degree it is possible to use other instruments (e.g. the PPO) in Danida to proactively monitor that PCRs are being submitted in time (i.e. not later than 3 months before project completion). An assessment has been made of the degree to which the projects for which PCRs have been received during 2006 has been shown in the Annual Report (the PPO annex) from Danida. In principle it would be expected that the projects, which have been completed during 2006, would be shown in the Annual Report 2006 as having been completed during that year. As this report was not available, it was instead tested to which degree the projects have been shown as ongoing in previous Annual Reports (from 1994 to 2005). Of the 315 projects covered by the PCRs received during 2006, a total of 253 projects have not been stated in any of the above mentioned Annual Reports. There is one major reason for this: The Annual Report only includes projects above DKK 5 million (before 2004 the limit was DKK 3 million). Except for 1 project16, all projects not shown are below this budget level. About 2/3 of the projects not shown are from the following sectors: Research, Emergency Assistance, Business and Good Governance. This is also a reflection of the fact that projects from these sectors are generally small. Country wise three countries (Tanzania, Nepal and Vietnam) account for a little more than 40 % of the PCRs not shown in the Annual Reports and which cover projects which can be assigned to a specific country (i.e. not regional or international projects). It has also been attempted to test what the expected number of PCRs for projects above DKK 5 million would be during 2006. Using the Annual Report of Danida for 200517 the test shows that a total of 42 projects with an official completion year between 2001 and 2004 were still ongoing at 16 File number 46.H.7-1-137. “Nødhjælpsoperationer i forbindelse med krigen i Irak gennem Dansk Røde Kors”. 17 Drafted at the beginning of 2006.

the end of 2005. Further, 182 projects18 were mentioned to have a completion year of 2005. Of these, 20 were mentioned as having been completed, i.e. there remain 162 programmes and projects, which were expected completed during 2005. A portion of these19 would submit PCRs during 2006, provided they were completed during the year. In addition, a total of 146 programmes and projects20 were mentioned in the Annual Report 2005 as having a completion year during 2006. A portion of these would be expected to submit a PCR during 2006. It cannot be seen from the Annual Report 2005 what would be the exact number of actual completed projects during 2006. However, deducting the 57 Mixed Credit projects mentioned in the Annual Report (PPO Annex) 2005, and based on the above analysis it can only be said that the maximum would be 29321. 63 PCRs for projects above DKK 5 million were received during 2006. Comparing the number of projects mentioned as completed in the Annual Report 2005 (the PPO Annex) and the list of PCRs received, it is seen that 5 projects22 are yet to submit PCRs. 4. LESSONS LEARNED 2006 Annex 2 of this report presents a condensed version of the lessons learned as expressed in the PCRs received during 2006. The wording in the PCRs has not been changed and no attempt has been made to make an interpretation of the statements as expressed in the PCRs. However, for each of the lessons learned a short sentence is shown besides (in italics), capturing the most important message. It should be mentioned that some of the lessons learned are debatable. However, they have been presented without attempting to make an interpretation or discussion of these. The lessons learned shown in Annex 2 have been selected from the point of view that they should have wider relevance than the project itself (e.g. as only providing an input to preparation of a new phase) and provide information that is useful for a wider audience. In the absolute majority of the cases the lessons learned shown in Annex 3 has specifically been stated as lessons learned in the PCRs. In a few cases the text in the PCRs has implicated that it was a lesson learned, although it was not specifically stated. The lessons learned have been gathered from all types of PCR reports, whether filled in according to the new or another format, as the lessons learned have been regarded as being useful for a wider audience. E.g. lessons learned from NGO projects have also been included although the NGO completion reporting format has been used. A goal has also been to keep the condensed version as short as possible. The lessons learned have been grouped according to major “sectors”, country specific learning and “subject matter” learning (evaluation, design and implementation approaches and management).

18 Excluding multilateral projects. 19 Except for the Mixed Credit projects. 20 Excluding multilateral projects. 21 293 = 42 (completion between 2001 and 2004) +162 (completion in 2005) +146 (completion in 2006) –57 (Mixed credit projects). 22 (i) 104.Namibia.1.MFS.12: “Demonstration af vedvarende energiteknologier i Gobaleb”, (ii) 6.U.1.: “International Committee of Red Cross (ICRC) generelt bidrag 2005”, (iii) 104.Eritrea.813-1.: “Støtte til uddannelsessektoren i Eritrea, pilotfase”, (iv) 104.Eritrea.805-2.: “Skovfrøcenter”, and (v) 104.Sydøstasien.1.MFS.3.: “Miljøprogram for Mekong kommissionen”.

5. PCR FORMATS In the analysis of PCRs it has also been recorded in which language the PCRs are filled in. The absolute majority (about 86 %) of PCRs are prepared in English. PCRs for projects in Spanish speaking countries are typically not prepared in Spanish, but in English or Danish. PCRs for projects in French speaking countries are mainly prepared in English or Danish. A significant number of PCRs prepared in Danish are for NGO projects, which follows a specific project completion format. In the new formats for PCRs for programmes and components it is difficult not to use the relevant international language, as the Chairperson of the Steering Committee will have to endorse the PCR. As far as format 3 is concerned, the Embassy or the Head of Department have filled these out. The analysis shows that this in all cases has been done in the relevant international language. It should be mentioned that the PCRs for projects in Portuguese speaking countries neither have been prepared in Portuguese nor in one of the three new formats. The PCR formats should also be created in Portuguese. In some cases a PCR has been filled in partly in Danish, partly in English. This applies both to the new format 1 and to the “other” format category. Table 19 Language in Which PCRs are Filled in. A little more than 1/3 of the PCRs have been filled in using one of the three new formats. Of these the absolute majority is format 3 for projects below DKK 5 million. This reflects the overall composition of projects, of which the major part is small projects. The high number in the category “other” is mainly due to two factors: The use of another format for NGO projects and the backlog of PCRs received in old formats. With 8 format 1 it would be expected that there would be at least 24 format 2 as a programme typically contains at least 3 components. The reasons for the low number of format 1 are that in some cases format 1 has also been used for projects (which do not contain components) above DKK 5 million, and in one case there has only been a PCR made at the programme level (Health programme in Zambia). It should also be mentioned that in some cases format 1 has been used for components. As described in section 3 the number of scores for components are higher following some adjustments made.

Language of PCRs

271

3

5 36

English French Spanish Danish

Table 20 Formats Used for PCRs. In general it is the impression that the new formats are an improvement compared to older formats. The main improvements are that the new formats make it more easy and less time consuming for the respective parties to prepare the PCR, it makes sure that the highest programme/project authority (typically a steering committee) read and endorse the PCR in case of a programme, component or project above DKK 5 million, it ensures that it possible to make a statistical analysis of PCRs on the basis of a consistent set of data in the PCRs as well as it links better with the ongoing monitoring of the support programmes. A consideration could be to ask – e.g. through an electronic questionnaire – the people who fill in the formats, if they have suggestions for improvements or amendments to the formats as well as to the PCR guideline. The time period during which the new PCR formats have been in use is short, and the experience in using these are therefore relatively limited. The findings are the following: • In some cases there appears to be confusion between the formats (format 1 and 2) in terms of

which one to use. • It is unclear what the basis for the rating is on the cross cutting issues (format 1), as for all four

cross cutting issues there are three sub-criteria and only a possibility for giving one overall rating covering all three sub-criteria.

• In some cases there is a discrepancy between the rating and the assessment given for a given criteria.

• The section on results has been difficult to use, as most of the programmes and components were started before budgets became output based. It has therefore not been possible to indicate budgets and expenditure figures for one key output per component (format 1) or for component outputs (format 2).

• The understanding of the concepts effectiveness and efficiency has been difficult. • The description of the concept of capacity building in the format has been difficult to apply. • The relevance of the lessons learned in format 3 is questionable, as the focus among others

should be on alignment and harmonisation. For the small projects these areas are usually not the focus and the understanding of these concepts is limited among the project implementers.

• Format 3 gives the possibility to make a rating on the fulfilment of project objectives. In some cases more than one objective exists for a project. In those cases it has in a number of instances been unclear if the ratings given were for a development objective or for one or more immediate objectives.

• A total of 39 NGO projects have submitted a completion report during 2006. NGO projects are at present using a separate format. This format is more comprehensive than the new PCR formats and include a number of specific requirements to information that must be given. This

Formats Used for PCRs

8 6

94

198

0

50

100

150

200

250

Format 1 Format 2 Format 3 Other

additional requirement for information could, if essential for the NGO project, be provided in the regular progress reporting23. The NGO format does not include the possibility to apply a rating as in the new PCR formats. Although the rating will be carried out by the NGO itself, it does in this respect not differ in substance from other projects as a PCR in principle is a self-assessment by the project management. Some of the cells to be filled in in the NGO format resemble the ones of the PCR format, but there are small differences (e.g. table 5 in the PCR format and table 12 in the NGO format). Further, the NGO format does not contain a specific section in which lessons learned are described. This makes it difficult to draw out the lessons learned from the more general descriptions presented in the NGO format for project completion. Hence, the NGO format for project completion does not fully provide the possibility for using the valuable experiences from NGO projects in the analysis of PCRs.

In terms of quality of the PCRs received during 2006, it can in summary be said that less than half of the ones that should have followed the new format have been filled in in one of these. It is expected that this picture will be very different for PCRs received during 2007 as the new format have now existed for more than one year and a major backlog of older projects have been cleared during 2006. The consistent use of the new format would also be expected to lead to removal of Danish as a language to be used in PCRs, enabling the partners to read and comment on PCRs, and in principle to be the ones writing the PCR. Specifically regarding filling in the cells of the formats it can be said that in general this has been done but as indicated above there have been difficulties in some areas (choice of format 1 or 2 in a few cases, table 5 on achievements of results, use of concepts like effectiveness, efficiency and capacity building). A general comment is that in some cases it is felt that too limited time has been used to fill in the PCR for smaller projects in order to make it a useful instrument for others. As this is the only assessment at the end of the project that is typically made, time should in all cases be allocated to fill in the PCR format and give an assessment of the degree to which the objective is reached and elaborate on the valuable lessons learned. It is recommended that: • The NGO projects should use the same formats (either format 1 or 3) for project completion

reporting as all other bilateral projects, which are required to submit PCRs. • The PCR guideline be edited in order to provide additional clarity to the users which have to fill

these in. 6. HARMONISATION AND ALIGNMENT EFFORTS AND DANIDA

REQUIREMENT FOR PCRs One of the advantages, especially seen from the point of view of the partner of the harmonisation and alignment agenda is that the number of reports to be produced to different donors will be reduced. To the degree that Danida through its sector programme support participate together with other donors in supporting government programmes it also becomes relevant to ask if there should be prepared a separate PCR for the Danida participation in a joint donor supported programme. To the degree that the partner has its own programme completion format it also needs to be asked if a reporting in such a format would be acceptable to Danida. This discussion should take into account the scale of the issue: It is not known what the requirements are by government partners or organisations for completion reporting. These requirements will most likely vary from country to country. During 2006, none of the PCRs 23 Other programmes, components and projects are also subject to regular progress reporting in which more comprehensive information can be given.

received have been prepared in a partner format. The scale of the issue will depend on the degree to which Danida is supporting government programmes together with other donors and the degree to which there would be components, which is covering a “Danida only” intervention. For almost all small projects (below DKK 5 million) this is not likely to be an issue, and the new format should be simple to fill in and not be much time consuming. Considering that it is a legitimate requirement that the Danida supported programme report to the Danish authorities it will be required that Danida receives a programme completion report. In those cases where one agreed to format does not exist for programme completion reporting it would be an option to use the Danida format for PCRs. Other formats should however also be accepted, provided that the same type of information is included and that it is agreed to by the parner and donors. In this respect it should be noted that the requirements for information in format 1 (for programme completion reporting) in no way is comprehensive or request information that would require additional analysis or data gathering apart from what would already be available with the programme management. The same applies to format 2 for component completion reporting which is even simpler than format 1. The only additional requirement would be the scoring of performance according to a number of criteria. The opportunity exists for the Danish Representation in the country to make the scoring as part of their normal monitoring responsibility, and provide these in the relevant PCR format. The explorative study has shown that in a few cases a completion report has been received from international humanitarian organisations, which does not provide any of the “normal” project completion reporting information. In those cases the relevant format should be filled in by the responsible department in Danida and uploaded in the PDB. 7. WIDER KNOWLEDGE SHARING OF LESSONS LEARNED The present procedures for preparation, submission and use of PCRs can briefly be described as follows: • The PCR is drafted by the Programme/Project management for presentation to the decision

making programme/component/project authority. • The respective steering committee and the Head of the Danish Representation subsequently

approve the PCR as far as Programme, Component and larger projects are concerned. The Head of the Danish Representation/Head of Department approves PCRs for the smaller projects. For NGO, humanitarian and research projects, the respective department in Danida approve the PCR.

• The Danish Representation/responsible department is responsible for uploading the PCR in the PDB and for informing the Evaluation Department by e-mail that the PCR has been uploaded. The Evaluation Department does not receive an electronic copy of the PCR, as this report at the time of notification exists in the PDB.

• The Evaluation Department keeps hard copies of the PCRs covering the period 1992 (when they were introduced) until the end of 2003. From 2004 onwards the PCRs are stored in the PDB, and as such they are in principle easily available for others to use.

It should be mentioned that the PCR, at the time of uploading in the PDB is marked as “Completed” by the one responsible for the uploading. However, this does not necessarily imply that the final accounts have been settled and the audit report approved24. This typically happens a number of 24 This seems to be different to the procedure described in the PCR Guideline (Evaluation Department responsibility, page 3).

months later (according to the new PCR guideline the final accounts and audit report must be submitted to Danida not later than 6 months after project completion). The final accounts are not uploaded in the PDB. It is recommended that: • The PDB is amended so that there are two fields to be marked for completion of PCRs: One

field for marking that the draft PCR has been uploaded and one field for marking that the final accounts have been approved and the project therefore can be regarded as also being financially completed.

It should also be considered to upload the final accounts and audit report in the PDB in order to have all reports for a programme/project in one location. It is the impression that PCRs is a very useful tool for the Evaluation Department as an important input into different evaluations. Much useful information is now readily available when preparing further detailed overview, reviews or evaluation of specific sector-wise or country programme activities. The PCRs contain lists of documentation, and indicate where further information can be found. As such, the completion reports are the main (and often the only) tool for tracing the longer-term results of the Danish supported development activities after termination of the support. It also seems safe to assume that important lessons learned from programmes and projects, as described in PCRs25, are utilised in design of a new programme/project phase in the country in question. With the responsibility for preparation of new programmes/projects being vested with the Embassy, the lessons learned should be part of the institutional memory at the Embassy. However, at present there is no formalised mechanism whereby the lessons learned from programmes or projects, at least not in the form of sharing PCRs, are transferred e.g. from one country to another or from Embassies to e.g. BFT staff which will have a role in appraisal of new programmes. Even though Danida staff has access to the PDB in which the PCRs are stored, the concerned staff may not know that a new and relevant PCR is available unless they check the PDB). A step to improve this would be to put the PCRs on the Danida Intranet in a systematised manner, e.g. according to country and/or sector. This would in an easy way provide access to PCRs to relevant staff. If this option is chosen, the Internal News column on the Intranet should inform staff that a group of PCRs (e.g. for one quarter) have been uploaded on the Intranet. An alternative would be to further improve the PDB enabling staff to easily search for e.g. PCRs according to sectors or countries. While the PDB contains the documents (PCRs), it does at present not have a search function enabling the individual staff member of Danida to easily seek this information. It requires that a contact is made to the PDB office or that the staff member knows the Danida file numbers of the projects/programmes. The information contained in PCRs is not confidential and the PCRs are not classified documents. Hence there should not be any reason for not publishing the lessons learned from PCRs. This could be done at two levels: • Publishing all PCRs in a systematised manner on the web site of the Evaluation Department (or

alternatively at the BFT web site), as an alternative to publishing it at the Intranet, and/or

25 The PCRs would in most cases only be drafted after a new phase has been prepared – but it is expected that there would be consistency between the lessons learned as described in the PCR and the design of the new programme/project.

• Publishing a condensed version of the lessons learned together with PCR statistics (see further in section 8).

The first option will provide the entire information contained in all PCRs to users (Danida staff, partners, NGOs, consultants, journalists and the public in general). However, the amount of information will be large and it will be time consuming for the individual to obtain an easy overview. The second option will cater for this concern. The final decision on publishing PCRs should take into account the overall communication strategy of the Ministry of Foreign Affairs. It is recommended that: • PCRs are presented in a systematised manner on the web site of the Evaluation Department for

reasons of transparency; • The condensed version of the lessons learned and statistics of PCRs are published on the web

site of the Evaluation Department; • The PDB is improved to include a user-friendly search function. 8. ANNUAL UPDATE OF LESSONS LEARNED AND PCR STATISTICS With the increased focus at PCRs and of dissemination of the results, it will be needed for Danida to prepare an annual update of the lessons learned as expressed in PCRs as well as of the statistics which can be generated from the received PCRs during the year. It is recommended that: • An annual update is made of the lessons learned as expressed in PCRs and of the statistics that

can be generated from PCRs. In order to secure a smooth preparation of the annual update and in the best possible quality, the methodology to be used in this process can be divided into two main parts: • A quality assurance procedure being applied through the year, each time a PCR is received, and • The process of preparing lessons learned and statistics on the basis of quality assured PCRs. The quality assurance procedure should include the following and be used each time a PCR is received: • It should be checked if the correct PCR format has been used. Use of an incorrect format should

lead to a request for preparation of a PCR in the relevant new format. In case the PCR is for a project that has been completed before the introduction of the new formats in January 2006, the new formats should also be used.

• It should be checked if all cells have been filled in and questions answered. This includes among others starting and ending dates, both planned and actual, approved budget and actual used budget, and the DAC code using the latest list from DAC.

• The PCR should be submitted in a Word file. The Evaluation Department or any other office in Danida to which this responsibility may be assigned, when receiving the notification e-mail about uploading of the PCR, undertakes a quality assurance of the PCR. At this time the PCR should be stored in a created folder, as this will avoid double work in relation to collection of all PCRs for the annual assessment. At the same time a hard copy of the Grant Page for the PCR as generated by the PDB is made and stored. Both of these activities can save time at the end of the year, as it will not be required to retrieve the PCRs from the PDB again.

The process for preparation of lessons learned and PCR statistics should be done on the following basis: • Enabling PCR statistics to be included in the Annual Performance Review the process should be

carried out within the period of 1st of December and 31st of Janaury; • All PCRs should be copied to a CD Rom and delivered to the one responsible for preparation of

lessons learned and PCR statistics; • All PCR files should named with both the PCR number and Danida file number; • A hard copy of the Grant Page as generated by the PDB for each PCR should be provided for

easy reference; • All PCRs uploaded in PDB during the calendar year should be included. PCRs received the

previous year should be excluded; • A list of all PCRs should be generated in Excel by the PDB office, including PCR number,

Danida file number, title, country and applicant grant. Only PCRs received during the calendar year should be included;

• Considering that there might be incoming PCRs during the month of December, it is needed that (i) a CD Rom with PCRs from the months of January to November be delivered by 1st of December together with the Excel list from PDB, and that (ii) a CD Rom with PCRs from December together with an updated Excel list from PDB is delivered by the end of December at the latest.

This procedure will enable lessons learned to be prepared on the basis of reading the majority of PCRs before the end of December. As PCR statistics is most easily prepared after all PCR data have been included in the “PCR database”, this part of the process is best done during the first part of January. Following the preparation of lessons learned and PCR statistics, an input can be prepared to the Annual Performance Report. The wider analysis of lessons learned and statistics can subsequently be uploaded on the Danida web site.

32

ANNEX 1

TERMS OF REFERENCE

33

Final Draft

initialer kontor journalnummer dato

HAN EVAL 104.A.1.e.58. 13 November 2006.

TERMS OF REFERENCE for

Explorative Study of Programme/Project Completion Reports 1. Background: When project-and programme completion reports, PCRs, were introduced by the late 80s the purposes of reporting at the stage, where Danida’s development support is terminated, were several:

- To document the status of the project (and of the results achieved) at the time of the completion of the assistance;

- To provide an assessment of the need for possible further assistance after completion of the project;

- To contribute to the accumulation and dissemination of experiences (lessons learnt); Finally, the PCR would also provide a basis for determining whether an evaluation (final or ex-post) is warranted. The first guidelines for the elaboration of the PCRs were drafted by a task-force in 1988/89, and tested up to 1991. Subsequently, the Head of the South Group 19 May 1992 directed all units and relevant embassies to draft a completion report of all Danish (bilateral) assistance activities, approved by Danida’s Board, and completed after 1 May 1992. Only those projects of which a PCR had been drafted could be included in the Annual Report to the Finance Committee (‘Projektorientering)’ as completed. A revised Guidelines for Project Completion Reports was issued in 1997, making a distinction between PCRs for projects of more than DKK 3 mio., and project completion sheets, PCS, of assistance between DKK 0,1 and 3 mio. Moreover, the general Sector Programme Support Guidelines of 1996 introduced the PCRs of SPSs and SPS-components. With the introduction of the electronic project database, PDB, in 2000, the regional departments were directed to upload the PCRs on the PDB, and subsequently from 1 September 2003, with the decentralisation of the development assistance, all representations were made responsible for the direct uploading of PCRs on the PDB in a standard format. Subsequently, the Evaluation Department is to be notified by the representation by e-mail when the PCR has been uploaded on the PDB, and no paper version is submitted. Since 1992 about 4.300 PCRs have been drafted, most of which are now uploaded in the PDB as an electronic copy. The Evaluation Department has maintained a physical archive of the received reports during the period 1 May 1992 – 1 September 2003, out of which at least 750 PCRs are still only available in paper versions. The documentation purpose of the PCRs has frequently been fulfilled. Much useful information has been utilised when preparing further detailed overview, reviews or evaluation of specific sector-wise or country programme activities. E.g. for the Pre-study of the evaluation of the Danish Uganda Country Programme including all Danish assistance to one country over a 17-year period the PCRs in such cases are indispensable.

However, to assess the quality of the PCRs, and in more general terms to deduct the lessons learned several attempts have been made to undertake analysis of the reports. As the quality, formatting and contents of the PCRs has varied substantially, introduction of systematic assessments on an on-going basis has been difficult. In addition, most PCRs are descriptive, and not very analytic. In 1998, a consulting company was commissioned to look into the PCR-system. Based on a sample of about 200 PCRs they found i.a. that many reports are drafted long time after the completion of the activities making them less useful; and that the standard review sheets are rarely used for comments on the drafted PCRs by the respective embassies and regional offices. They recommended in particular to simplify the reporting formats, and to introduce a graduation of the achievement of the support objectives. More information should be provided by simply ticking pre-designed options to facilitate the subsequent assessment and statistical analysis. By 1 January 2006 a new format was introduced, partly complying with the previous recommendations, and in particular to better link with the on-going monitoring of the support programmes and the regular performance reviews. The new format and guidelines of the PCRs are now available from the Aid Management Guidelines, and apply to all bilateral programmes and projects in both programme and non-programme countries. A PCR must be prepared at the end of the appropriation period (i.e. at the end of each phase of a programme) by the programme/ project management, and presented to the decision-making representatives of Danida and partners (e.g. the Steering Committee) three months before the end of a programme, project or component. After the accounts have been finalised and audited, the final version of the report must be approved by the relevant partner representative and the Head of the Representation/Department at latest six months after the end of the programme, project or component, and entered in the PDB by the responsible desk-officer of the representation (or, in the case of non-programme countries, by the regional department). Only three formats are now used: A Project Completion Sheet for projects below DKK 5 million; a Component Completion Report for components; and a Programme/Project Completion Report for programmes and projects above DKK 5 million. To follow-up and explore the application of the new programme- and projects completion-system an explorative desk-study of the current status as per end 2006 is now to be undertaken to provide the necessary basis for decision of the further development of the PCR-system. 2. Purpose: The overall purpose of the study are:

- To contribute to the establishment of a simple system of (scheduled/annual) electronically based and user-friendly synthesizing of lessons learned from PCRs;

- To contribute to the development of a systematic and on-going knowledge-sharing of Danish lessons learned among MFA-staff, partners in programme countries and non-programme countries, and for the wider public, accessible via the internet and/or the intranet;

3. Scope of Work: The consultant’s assignment will comprise, but not be limited to:

- Provide and update overview of current contents, distributed as per time periods received; country/regional programme; sector; NGO, Gov’t, multilateral, etc.; specify in particular the PCRs received in the new format up to end of 2006;

- Compare with other lists of projects/programme (PDB, annual project/programme information, Annual Report of Danida, list of Auditor-General), and identify gaps, if any;

- Carry out a quality check of received PCRs after 1 January 2006: to which degree have the formats been properly filled in?

- Identify common users’ constraints/challenges relating to current format; - Consider how the demands and requirements of Danida of PCRs are compatible with the

harmonization and alignment efforts in the fields of programme- and project reporting, and advise to what extent a PCR prepared according to a partner’s own format be accepted by Danida;

- Analyse and compare the new filled-in format with the previous filled-in formats, and present differences, strengths and weaknesses;

- From the new PCRs: provide an overview of the responses question by question, synthesize and analyse the responses, and present the findings in simple matrixes and boxes; if necessary, carry out the analysis and synthesis as a pilot for only a few countries, and for a few sectors;

- Identify those PCRs received after 1 January 2006 which are not drafted in the new format; - Register the time used for the elaboration of the overview, analysis, synthesis, and

presentation of the new PCRs; - Assess how the PDB is functioning in relation to the filing, access, reporting, etc. of PCRs; - Recommend how a simple, annual exercise of updating the synthesis could be undertaken,

primarily as an electronically based and user-friendly approach; - Recommend how a wider knowledge-sharing of the lessons learned of the PCRs could be

organised, mainly accessible via the internet and/or the intranet; - Carry out any other task related to the study as agreed upon; - Draft a brief report (max. 10 pages) of the work carried out.

4. Outputs: The outputs of the assignment shall be:

- As CD-Rom: Tables/lists of the status of PCRs received, missing, analysed, etc. - Weekly short status reports with time consumed (max. one page). - Brief report including the findings and recommendations, and the work carried out (max. 10

pages), annexed with the CD-Rom. 5. Required Inputs: Consultant, qualified in data collection and analysis, desk-studies, knowledgeable and experiences of development issues and Danish development policies, administration and guidelines; available for internal desk-work in MFA for 250 hours over 12 weeks November-January 2006. 6. Timing: As soon as possible, beginning by end October. Delivery of draft final Report by 5 February 2007. 7. Reference: The consultant will work closely with and be guided in the work by the responsible desk-officer of the Evaluation Department, Senior Technical Adviser, Henrik A. Nielsen. The consultant will not have direct access to the PDB, but information and reporting from the database will be provided by EVAL’s Secretary, Ms. Else Duch. Dato BFT sagsbehandler (underskrift/navnestempel)

ANNEX 2

LESSONS LEARNED

ANNEX 2 LESSONS LEARNED Water, Sanitation and Hygiene Sector Vietnam, Support to Small-Scale Private Sector Development and Marketing for Sanitation in Rural Areas: External subsidies for business development and promotion is often more sustainable than subsidizing sanitation hardware

Use of external subsidies for business development and promotion is often more sustainable than subsidizing sanitation hardware, because once demand is stimulated and the market is established, suppliers take over promotion even if external funds are withdrawn. This also means that the consumer does not have to experience a sudden price shock from unsubsidised hardware, which has led to the collapse of many past sanitation programmes.

Bangladesh, Arsenic Mitigation Component: Many people continued using arsenic contaminated water when there was no safe water source nearby

“The tube well screening operation was not welcomed by a significant portion of the people because when their tube wells were found to be arsenic contaminated and red marked, no alternative source of safe water was provided. Therefore many people continued using arsenic contaminated water when there was no safe water source nearby and preferred water of red marked tube wells above polluted pond water.” haring has created a sense of ownership. However, cost sharing policy should be developed on the basis of the socio-economic condition of the target group.”

Bangladesh, Rural Water Supply and Sanitation Component: Rural people always give priority to safe water rather than sanitary latrine

“The rural people always give priority to safe water rather than sanitary latrine. So sanitation promotion supported by water supply is a good strategy for shifting people’s attention to sanitation” “The interactive process of hygiene promotion is more effective than unilateral message dissemination process to bring desired changes in the behavioural pattern. Too many messages diluted community attention in hygiene promotion. Continued follow-up with limited messages gives better result and sustains a positive behavioural trend. Participatory approach for hygiene promotion especially for promotion of household sanitary latrine is very effective” “Only training and orientation do not build capacity and develop human resources unless the institutions and stakeholders are directly involved with the programme implementation”.

Bangladesh, Urban Water Supply and Sanitation Component: Institution Development and capacity building activities should be integrated with the hardware activities

“Institution Development and capacity building activities should be integrated with the hardware activities” “The water tariff must include the full Operation and Maintenance cost and depreciation for future investment and expansion programmes. The principle of depreciation should be understood and depreciation cost must be added to determine the water tariff”.

Bangladesh, NGO Forum Component: Hard core poor people require subsidy

“People’s participation in planning and management of WatSan programmes and facilities can be very much effective to achieve sustainability of programmes” “Hard core poor people require subsidy.”

Bangladesh, Water Supply and Sanitation Sector, Programme Completion Report: Substantial commitments from relevant institutions, substantive inputs in capacity building, and, not least, time required for sustainability

While the assessment is that the Government, in particular in the last 2-3 years, has taken a greater interest in, and responsibility for, programme implementation, the reality has been that much of the day-to-day programme implementation tended to remain in the hands of Danida-funded project implementation units and Danida Advisers. The recognition that that approach will not be sustainable in the long run, has led to the general acceptance that things will have to be done differently in the next phase. Considering the existing, limited, capacity in many of the partner organisations (at all levels), combined with the relatively limited experience of these organisations in managing some of the envisaged software/support activities, the expectation is that this will require substantial commitments from relevant institutions (e.g., in staff inputs), substantive inputs in capacity building, and, not least, time.

Education Sector South Africa, Support to Education and Skills Development (SESD), National Component: Lack of co-operation between two ministries – which both agreed to the programme document - hampered implementation

SESD I was formulated on the assumption that one programme document could cover support to and collaboration with two ministries. Although both ministries had agreed to the programme document, actual implementation was hampered by a lack of co-operation between the ministries.

Angola, Basics of educational assistance, by Save the Children: Unexpected positive effect was the level of satisfaction and joy brought to the teachers and headmasters

One of the unexpected positive effects of this project was the level of satisfaction and joy, which the project brought to the teachers and headmasters in distant villages all over the two municipalities. From being an unknown teacher in an isolated area to be in the centre of a project with innovations and support was a great motivation for the teachers, both the ones with many years of experience and the newly employed.

Iraq, Emergency Education and Child Protection Project by Save the Children: Education programmes take TIME!

The most important lessons learned through implementation include that education programmes take TIME! And also despite of the fact that some might consider education not to be an emergency intervention it is still crucial to build a better future for the children and their families.

Uganda, Special Needs Education / Educational Assessment and Resource Services: Ownership and integration of special needs education into the ministry’s structure at all levels necessary

The ownership and integration of special needs education into the ministry’s structure at all levels is necessary to achieve effective administration of the programme, close supervision, and accountability.

Health Sector Zambia, Health Sector Programme Support (HSPS) III: Alignment to national health policies was successful but some parallel management methods and procedures still exist In practise none of the advisors worked as “advisors” but as an extra pair of hand

“The Programme’s development objective, immediate objectives, expected outputs and activities were in line with policies, and so were most, but not all of its management methods and procedures” “The Health Sector in Zambia is one of the most advanced in terms of harmonisation, though parallel structures and organisations still exist and in some areas are on the increase” “Because of the programme’s embedment in the Central Board of Health (CBoH) structure at both national and provincial level, and the adoption of CBoH methods, the day-to-day responsibility for several programme management issues was with CBoH staff, which is undoubtedly very positive” “Experience has also shown that several HSPS III activities would not have been implemented without advisors at national and provincial levels. In practise, none of the advisors has been able to work as “advisor”. All worked as an extra pair of hand, i.e. as technical assistant. Despite restraint by all to do so, this was often necessary, as few national counterparts - most often at the level of specialists – were willing/able to take the lead”

Nepal, HIV/AIDS and Trafficking Awareness Education Programme: Awareness programme on HIV/AIDS and Trafficking is really useful

Awareness programme on HIV/AIDS and Trafficking is really useful in raising awareness about such a sensitive issue like Trafficking and HIV/AIDS among people of grass root level who are the real victim. Such type of programme helps in uniting them together to fight against such issue and lead to long term development objective i.e. stop increasing the prevalence of HIV Infection.

Agriculture, Fisheries and Forestry Sector Vietnam, Fisheries Sector Programme Support (FSPS), Programme Completion Report, which includes lessons learned at component level: The FSPS was often used to demonstrate effectiveness of development aid and used as a good example of development aid making a difference

“Although the FSPS was closely aligned with national programmes within the fisheries sector the initiation of a full fledged sector programme with Ministry of Fisheries did take more than one year to find its modus operandi. However, once the FSPS was under way, cooperation developed into a solid partnership” “Due to the close alignment with the sector priorities on ensuring sustainable economic growth the support was well received and good results achieved. The FSPS I was often used to demonstrate effectiveness of development aid and used as a good example of development aid making a difference” “A clear monitoring and evaluation system should be established which would, among other aspects, collect regular formal feedback from training participants, and conduct follow up tracer surveys. The above recommendations serve the purpose to move the attention of those implementing the programme away from immediate output focus towards outcomes and ultimate benefits” “Seafood Export and Quality Improvement Project (SEAQIP)