development effectiveness - undpweb.undp.org/evaluation/documents/final_development... ·...
TRANSCRIPT
REVIEW OF EVALUATIVE EVIDENCE
EVALUAT ION OFF ICE SEPTEMBER 2000
DevelopmentEffectiveness
DE
VE
LOP
ME
NT
EF
FE
CTI
VE
NE
SS
- R
EV
IEW
OF
EV
ALU
ATI
VE
EV
IDE
NC
E
2
The views expressed in this publication do not necessarily represent those of the member counties of the UNDPExecutive Board or of those institutions of the United Nationssystem that are mentioned herein. The designations and terminology employed and the presentation of material do notimply any expression of opinion whatsoever on the part of the United Nations concerning the legal status of any country,territory, city or area, or its authorities, or of its frontiers or boundaries.
Copyright © 2000United Nations Development ProgrammeOne United Nations PlazaNew York, NY 10017, USA
www.undp.org
Information from this report may be freely reproducedso long as credit is given and tear sheets are provided to: Evaluation Office/UNDP
Book Design: InMind Design
Printed by means of environmentally compatibletechnology on recycled paper
TAB
LE O
F C
ON
TEN
TS
3
Table of Contents
Acronyms 5Introduction 6I.Assessing Development Effectiveness: Managing for Results 8
Results-based Management Reforms 8Results-based Management and Development Effectiveness 9Evaluating Development Effectiveness 10Beyond the Numbers: Understanding Development Effectiveness 13
II. Uncovering UNDP Development Effectiveness: Evaluative Findings 14Strategic Focus and Comparative Advantage 14Partnership 16Ownership 17Capacity Building 19
III. Unlocking Development Effectiveness: Current Mechanisms andDirections for the Future 23Institutionalising Results-based Management 23Strengthening Substantive Accountability 24Methodologies for Assessing Impact and Improving Feedback 25Linking Evaluation Recommendations to Decision-making 26Acquiring Knowledge and Sharing Lessons: the Value of Partnerships 28Strengthening Evaluation Capacity Development 30
IV. Conclusions 32Glossary 33Annexes 35
I. Trend Analysis: Data and Methodology 35II. Review of the Sub-regional Resource Facilities (SURFs) 36
Figures1. Achievement of Immediate Objectives 102. Achievement of Immediate Objectives by Quality of Project Design 113. Project Sustainability 124. Achievement of Immediate Objectives by Level of National Ownership 135. Quality of Project Management for NEX and Non-NEX Projects 186. Achievement of Outputs for NEX and Non-NEX Projects 187. Achievement of Immediate Objectives for NEX and Non-NEX Projects 188. Sustainability for NEX and Non-NEX Projects 19
4
TAB
LE O
F C
ON
TEN
TS Boxes1. RBM: A Shift in Mindset 82. Strategic Results Frameworks 93. Roles of Partnership 164. Sri Lanka: Moving from Concepts to Practice 225. Institutionalising RBM: Lessons Learned 236. Effective Follow-up of Recommendations 267. Following through on Evaluation Findings – Decentralisation 278. Practitioner-oriented Information – Essentials 29
5
AC
RO
NYM
SAcronyms
ADB Asian Development Bank
APR Annual Project/Programme Report
BDP Bureau for Development Planning
BMZ German Ministry for Economic Cooperation and Development
CEDAB Central Evaluation Database
CGAP Consultative Group to Assist the Poorest
CLIA Country Level Impact Assessment
DFID Department for International Development (United Kingdom)
ED Entrepreneurship Development
ECD Evaluation Capacity Development
EO Evaluation Office
ERP Evaluative Research Programme
EVALNET Evaluation Network
MDGD Management Development and Governance Division
MYFF Multi-Year Funding Framework
NEX National Execution
NHDR National Human Development Report
OECD/DAC Organization for Economic Cooperation and Development/
Development Assistance Committee
PEIS Project Evaluation Information Sheet
RBM Results-based Management
ROAR Results-oriented Annual Report
Sida Swedish International Development Agency
SME Small and Medium Enterprise
SRF Strategic Results Framework
SURF Sub-regional Resource Facility
UNDP United Nations Development Programme
UNOPS United Nations Office for Project Services
WB World Bank
6
INTR
OU
DC
TIO
N
The ability to promote concrete improvementsin people’s lives lies at the heart of what istermed development effectiveness. In recentyears, while there is impressive convergence oninternational development goals, there is equal-ly growing interest in seeing aid agenciesdemonstrate more effectively the purpose andvalue of their interventions. How effective aredevelopment projects and programmes inchanging people’s lives? How do aid agencies,in coordination with their national partners,maximise the benefits from taxpayer resources?More generally, donors, stakeholders and tax-payers have a collective interest in understand-ing better the use of aid resources and the dif-ferences these resources are making in the livesof people. In other words, what works, and forwhom?
This report represents an attempt to respond tosome of these questions for UNDP. Drawingupon evaluative evidence, it seeks to respond tothe evaluator’s perennial "so what?" question:"The organization may have made impressiveprogress but what does that progress amount toin terms of improving development conditions?"
The report assesses current challenges andobstacles facing UNDP and other developmentinstitutions in achieving development effective-ness. It seeks to highlight the various compo-nents of development effectiveness and todefine the role and particular contributionmade by UNDP. While the current informationbase does not yet permit a comprehensiveanalysis of all dimensions of development effec-tiveness, it provides a useful complement toexisting mechanisms for assessing the organiza-tion’s performance. In taking a medium- tolong-term perspective and presenting an in-depth, evaluation-based examination of the ele-
Introduction
ments that determine development effective-ness, this report complements the analyses pro-vided in the UNDP Results-oriented AnnualReport (ROAR).
A wide-ranging process of reform is underwaythat seeks to enhance UNDP developmenteffectiveness. At the core of this reform processhas been the introduction of results-based man-agement (RBM). By setting out clear andmeasurable goals, RBM marks a clear depar-ture from the traditional preoccupation withinputs and places new emphasis on tangibledevelopment results.
The adoption of RBM also highlights theessential role of monitoring and evaluation. Thechallenge to demonstrate value for moneyrequires the ability to accurately monitor, assessand evaluate the outcomes of development pro-grammes and projects. In this context, by pro-moting accountability and transparency, evalua-tion is increasingly recognised as a central com-ponent of good governance and an integral partof responsible civil action. As emphasised dur-ing the UNDP-sponsored Beijing Conference,"evaluation has become more important, notless," in enhancing development effectiveness.
Chapter 1 assesses how results-based manage-ment is contributing to improving UNDPdevelopment effectiveness. It presents the over-all RBM framework, the current mechanismsupon which it relies, and its symbiotic relation-ship with the evaluation and monitoringprocess. A quantitative analysis of UNDP per-formance, including trends over time, provides apreliminary picture of the organization’s contri-bution to development effectiveness. The pic-ture it reveals includes encouraging indicationsof increased effectiveness over the past decade.
7
INTR
OD
UC
TIO
N
Equally important, however, are the deeperquestions raised by this quantitative analysisand by the conclusions of the Results-OrientedAnnual Report (ROAR) concerning the under-lying factors determining development effec-tiveness.
Based on evaluative findings, Chapter 2explores these questions in greater depth. Itanalyses the various challenges associated witheach issue, from the topic of organizationalstrategy to the problems of capacity building.Preliminary emerging lessons and best practicesare examined alongside the UNDP role andcontribution. Emphasis is placed on the con-crete measures that need to be taken in order tostrengthen development effectiveness.
Chapter 3 examines current responses to thesechallenges, highlighting a number of mecha-nisms aimed at unlocking development effec-tiveness. Recognising that these reforms consti-tute the first steps only, this section seeks toprovide the groundwork for future endeavours,both from an institutional and strategic view.The need for tighter and more systematic link-ages between evaluative findings and decision-making is brought into focus. In particular, thereport identifies shortcomings in following upon evaluation recommendations at both coun-try and corporate levels. The importance ofstrategic partnerships for results is recognised,as well as the need for consolidation and shar-ing of knowledge. Finally, the chapter assessesthe role played by evaluation capacity develop-ment in underpinning the development effec-tiveness of national partners.
Limitations in the data available demand a cor-responding modesty in their presentation.While evaluation findings provide valuable per-spectives on UNDP development effectiveness,they do not tell the whole story, due to theuneven depth and coverage of evaluations, their
project orientation and the detailing of pastrather than current achievements. Understandingand assessing the full impact of developmentinterventions requires a commitment to gobeyond individual findings, challenge oldassumptions and uncover new linkages amongan ever-increasing number of variables. Assuch, it is a work in process, in which consulta-tion, partnership and the sharing of new ideasand lessons play a vital role. Ultimately, thisreport presents the contribution of UNDP tothis learning process.
Khalid MalikDirectorUNDP Evaluation Office
8
I. A
SS
ES
SIN
G D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
MA
NA
GIN
G F
OR
RE
SU
LTS
In recent years UNDP has underpinned itscommitment to development effectiveness withintensive efforts to increase accountability andstrengthen the assessment of its interventions.This focus has required a major shift in mind-set and a strategic reorientation – changes thatlie at the heart of ongoing reforms – to intro-duce results-based management.
This chapter identifies the challenges that gaverise to results-based reforms and analyses thepotential benefits in terms of greater strategicfocus and effectiveness. It presents the variousmechanisms upon which RBM relies, includingthe pivotal role of monitoring and evaluation inpromoting development effectiveness.
Results-based Management Reforms
The notions of results and "managing forresults" are not concepts new to UNDP. Theorganization has been working to get thingsdone and produce results for many years, but byfocusing heavily on managing inputs and activ-ities and devoting less attention to the conse-quences of its actions. The corollary of this hasbeen that UNDP has not always been able toeffectively demonstrate its results to the full sat-isfaction of donors and stakeholders. Thechanges summarised here show the move inemphasis from inputs to outputs and outcomes,with the latter achieved through an outwardly-oriented, partnership approach (see Box 1).
Overall, UNDP has taken significant steps toput into place a system that sets out clear sub-stantive goals for the organization and includesindicators to monitor and assess progress inmeeting them. Through the introduction of
results-based management, it is trying toanswer more fully the basic questions surround-ing aid institutions and their effectiveness.With regard to both domestic and internation-al agencies, the concerns of taxpayers arestraightforward: they want to know to what usetheir resources are being put, and what differ-ences these resources are making to the lives ofpeople. These are difficult but necessary ques-tions that require appropriate responses fromthe aid community.
UNDP has seen the RBM exercise as an oppor-tunity to deepen an internal process of manage-ment reform begun in 1997 with the UNDP2001 change initiative. It recognised that thesechanges were necessary because of a shift in theexternal environment and the new andincreased demands being placed on the UN,particularly UNDP. In the new environment,UNDP becomes less an institution transferringresources and more a focused developmentagency. This focus prompts the organization toact proactively rather than reactively, and it laysstress on the results attained rather than thevolume of assistance provided.
I. Assessing Development Effectiveness: Managing for Results
BOX 1: RBM: A SHIFT IN MINDSET
FROM TO
Entitlements ResultsControl DelegationInputs Outputs/OutcomesEx ante Ex postBureaucratic SpeedRisk averse Opportunity drivenIntrovert Partnerships
I. A
SS
ES
SIN
G D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
MA
NA
GIN
G F
OR
RE
SU
LTS
9The challenge for UNDP has been to fashionan approach to RBM that stresses results andgreater focus, without sacrificing the organiza-tion’s commitment to decentralisation andresponsiveness to country needs and priorities.
Results-based Management andDevelopment Effectiveness
A key contribution of the RBM exercise, recog-nised from the start, is the enhanced ability itaffords UNDP to identify what it does best anddetermine precisely where it can best add valuewithin the broad areas of poverty, governance,the environment and gender. A fifth pro-gramme category covers special developmentsituations, encompassing support provided byUNDP in response to disaster- and crisis-relat-ed events. In addition, a sixth (non-pro-gramme) category of assistance, support to theUnited Nations, is provided by UNDP in itsrole as the custodian of the UN ResidentCoordinator function. By setting clear measur-able goals, the results-based approach seeks topinpoint the areas in which UNDP is success-ful in each category, as well as the areas whereimprovements are needed. This is achievedthrough the mechanism of Strategic ResultsFrameworks (SRFs).
The SRF is designed to be the primary plan-ning tool for country, regional and global pro-grammes and to serve as a management tool forboth headquarters and the country offices (seeBox 2). The individual SRFs are broughttogether to form the multi-year funding frame-work (MYFF). The MYFF is also the basis fora funding compact with the Executive Board,whereby UNDP has committed itself to estab-lishing a four-year framework based on the keyresults identified and targeted by the organiza-tion.
The MYFF represents a major change forUNDP since, for the first time, a comprehen-sive framework clearly states the organization’splanned results and provides an integratedresource plan that incorporates and presents ina transparent manner all financial allocationscovering programmes, programme support andadministrative operations. It is based on thepremise that public institutions can no longerlay a claim on public resources on the basis ofmandates alone. Rather, they have to outlinespecific programmes and services and funda-mentally demonstrate impact.
Based on the intended results outlined in theMYFF, the Results-Oriented Annual Report(ROAR) is the principal instrument providing asystematic analysis of the actual results achievedby UNDP, and showing how these results relate
BOX 2: STRATEGIC RESULTS FRAMEWORKS
To bring added definition to the intended results of UNDP’s diverse activities and to concentrate UNDP’s efforts around the organization’s mandate and comparative advantage, Strategic Results Frameworks (SRFs) define for each area of focus:
(a) a clear goal, normally building on global conferences supported by the UN and inter-agency agreements (such as those of theUN Administrative Committee on Coordination);
(b) sub-goals (typically two or three) emphasising UNDP’s distinctive contributions; and, for each sub-goal,
(c) several strategic areas of support reflecting UNDP’s areas of comparative advantage and programmatic emphasis.
Country offices and other operating units, working within this framework, identify the particular outputs and outcomes they expectto achieve, set specific targets, and subsequently report on accomplishments in relation to these intended results.
I.
AS
SE
SS
ING
DE
VE
LOP
ME
NT
EF
FE
CTI
VE
NE
SS
: M
AN
AG
ING
FO
R R
ES
ULT
S
10to the six goals of the organization. The ROARpresents a comprehensive picture of whereUNDP has focused its efforts and analyses theprogress made against the intended outcomesset out in the MYFF. As such, it provides botha critical opportunity to test and streamline theapplication of results-based management inUNDP and an empirical basis for assessingUNDP development effectiveness, thus permit-ting a closer alignment of current and futurework o f UNDP wi th the v i s ion o f theAdministrator, set out in his Business Plansfor 2000-2003.
As a consequence of the energies devoted to thedevelopment of strategic results frameworksand the ROAR, the emphasis on developmenteffectiveness has been strengthened withinUNDP. At its core lies a focus on outcomes,which is still new to the organization and needsto be strengthened so that it becomes a normalpart of its understanding and work. At thecountry level, this will require commitment tostrong monitoring and evaluation of key out-comes, for which there is a demand for UNDPto contribute, in partnership with others. At thecorporate level, UNDP is currently taking stepsto simplify the SRF and render it more strate-gic. Success in both areas is central to theimprovement of development effectiveness.
Evaluating Development Effectiveness1
Successful implementation of RBM is heavilydependent on feedback and evaluation.Without a system in place to provide feedbackon results, benchmarks cannot be produced andlearning cannot take place. In this context,evaluation reports provide an important sourceof evidence through which UNDP substantiveperformance can be assessed.
The currently available evaluative data is large-ly project-oriented. It is important to recognisethat, while this project-level assessment ofeffectiveness provides valuable evidence on per-formance, it provides only a partial picture ofthe UNDP overall contribution to developmen-tal change at the country level. This gap isaddressed by focusing on outcomes and resultsthat go beyond individual projects – onUNDP’s "soft interventions," which are dis-cussed in the following chapter. Keeping thesecaveats in mind, one finds nonetheless thatthese tools yield interesting insights into thecomponents of UNDP performance that help itto meet its objectives.
Evaluative data indicate that UNDP has shownsignificant improvement since 1992 in majorareas denoting performance and success.Satisfactory or better achievement of projectoutputs has risen from 82 to 89 per cent andsatisfactory or better achievement of immediateobjectives has increased from 81 to 89 per cent(see Figure 1).
1 This section draws upon evidence of development effectiveness contained in CEDAB, see Annex I.
0 %
10 %
20 %
30 %
40 %
50 %
60 %
70 %
Significant Satisfactory Poor
1987-1991 1992-1998
FIGURE 1: ACHIEVEMENT OF IMMEDIATE OBJECTIVES
(PERCENTAGE OF ALL TERMINAL AND EX-POST EVALUATIONS)
I. A
SS
ES
SIN
G D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
MA
NA
GIN
G F
OR
RE
SU
LTS
11The type of analysis presented above also allowsfor some comparison of effectiveness acrossdevelopment agencies. This comparison can beproblematic given the fundamental differencesin agencies in terms of their portfolios and theirmethodologies for evaluations or ratings sys-tems. While keeping these qualifications inmind, some basic comparisons can be made,particularly by emphasising trends in perform-ance rather than absolute percentage levels citedby agencies.
Comparisons were possible with both theDepartment for International Development(DFID) of the United Kingdom and the WorldBank. DFID also shows an upward trend from66 per cent (1979-1988) to 73 per cent (1989-1992) to 75 per cent (1993-1997) of projectsrated as satisfactorily or better in terms ofachieving their immediate objectives. TheWorld Bank measures reveal an upward trendfor satisfactory or better achievement withrespect to outcomes, with 72 per cent in 1990-1993, 77 per cent in 1994-1997 and 81 per centin 1998-1999. The World Bank’s methodology,however, differs in a few significant ways in thatit groups projects by exit year instead ofapproval year (as UNDP and DFID do) andfocuses on outcomes as opposed to immediateobjectives.
These comparisons allow us to see UNDP inthe context of a broader community. While theyare not perfect, they do reveal that UNDP, likeDFID and the World Bank, appears to bedemonstrating an upward trend in perform-ance. To determine what the trends are able tosay about the overall development effectivenessof any of the organizations we must see them inthe wider context of the picture emerging fromthe monitoring and evaluation systems and theinformation generated by RBM. A range of fac-tors influencing UNDP development effective-ness is discussed below.
Project Design
Evaluators rate the quality of project design as away of judging whether UNDP plans and envi-sions its projects well both in terms of the coun-try context and the ability of the organization todeliver on its plans. In this area UNDP seems tobe performing well and improving, with 85 percent of all projects approved during 1992-1998having satisfactory or higher ratings for projectdesign, up from 77 per cent for interventionsdesigned earlier.
Furthermore, there is a strong positive correla-tion between high-quality project design andthe achievement of immediate objectives, as isclearly shown in Figure 2, below. This indicatesthat 67 per cent of projects which had a verygood quality of project design significantlyachieved their immediate objectives, whereasunder 5 per cent of projects with poor projectdesign significantly achieved their objectives.
0 %
10 %
20 %
30 %
40 %
50 %
60 %
70 %
80 %
90 %
100 %
V. Good Good Satisfactory PoorQuality of Project Design
Satisfactory PoorSignificant
perc
enta
ge o
f im
med
iate
obj
ectiv
es a
chie
ved
FIGURE 2: ACHIEVEMENT OF IMMEDIATE OBJECTIVES
BY QUALITY OF PROJECT DESIGN
0 %
10 %
20 %
30 %
40 %
50 %
60 %
70 %
80 %
Yes Partially No
1987-1991 1992-1998
I. A
SS
ES
SIN
G D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
MA
NA
GIN
G F
OR
RE
SU
LTS
12Other Measures of Good Performance
There has been a sharp increase in efficiency,defined as "the optimal transformation ofinputs into outputs,"2 with 56 per cent of allprojects rated as efficient after 1992 comparedwith only 40 per cent before 1992. The portfo-lio shows a marked improvement in the cost-effectiveness of projects since 1992, with 60per cent of the projects being rated as cost-effective, compared to 40 per cent before 1992.However, one-third of UNDP-supported proj-ects is still rated as partially effective or noteffective and, while this percentage is lowerthan in 1991, it remains high.
The share of evaluated projects effecting signif-icant institution-building increased from 22per cent in 1992 to 45 per cent in 1998, there-by signalling the success with which UNDP hasworked with its partners to promote thestrengthening of institutions. Significantimprovement can also be seen in the sustain-ability of projects. From 32 per cent before1992, the share of projects achieving sustain-ability has increased to 68 per cent since that
year. The UNDP Handbook for ProgrammeManagers defines sustainability as “the durabili-ty of positive programme or project results afterthe termination of the technical cooperationchannelled through that programme or project”.This clearly must be seen as constituting a keycomponent of development effectiveness.
Impact on Gender and Environment
Project impact with respect to promoting envi-ronmental sustainability and gender equalityremains a great concern. The ambition ofUNDP to mainstream environment and genderin all its interventions suggests that positiveimpacts would be expected from the majority ofprojects. Evaluators, therefore, have been askedto reflect upon impact in these two areas.While 46 per cent, up from 34 per cent, of proj-ects were rated as having a positive impact onthe environment since 1992, only 37 per cent,down from 46 per cent, were seen to have a pos-itive impact on gender since that year. This lastdrop is quite significant and is further compli-cated by a large percentage of projects – 53 percent for environment and 60 per cent for gen-der – rated as having no impact at all. Thesefindings are particularly striking given the gen-eral improvement of UNDP project perform-ance. They raise important questions about theeffectiveness of mainstreaming in UNDP andthe ability of projects to impact issues they donot specifically target. It is also important toconsider the extent to which these disappoint-ing results may also stem from inadequacies inthe existing mechanisms for assessing environ-ment- and gender-related changes. This is asubject of close scrutiny in the current refine-ment of the strategic results framework.
2 UNDP, A Handbook for Programme Managers: Results-oriented Monitoring and Evaluation, OESP, 1997. p. 26.
FIGURE 3: PROJECT SUSTAINABILITY
(PERCENTAGE OF ALL TERMINAL AND EX-POST EVALUATIONS)
I. A
SS
ES
SIN
G D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
MA
NA
GIN
G F
OR
RE
SU
LTS
13Assessing the Impact of NationalOwnership
Ownership of the programmes and projects bynational governments is increasingly recognisedas paramount to the success of an intervention.Evaluation findings show a strong correlationbetween the level of local ownership and theperformance of a project. Increased local sup-port for projects, due to higher buy-in and own-ership of the project by the national govern-ment, is one attribute that contributes to theimprovement of a project’s performance (interms of achievement of immediate objectives).
Given the correlation between the different lev-els of national ownership and the different lev-els of achievement of immediate objectives(Figure 4), greater national ownership of proj-ects can be seen to have a highly significantimpact on the success of interventions support-ed by UNDP. Figure 4 shows that 60 per centof projects with very good levels of nationalownership significantly achieved their immedi-ate objectives, compared to under 5 per cent ofprojects with poor levels of national ownership.
Beyond the Numbers: UnderstandingDevelopment Effectiveness
In recent years, a number of significant organi-zational changes have been instituted orlaunched. Among these are the programmeapproach, national execution, results-basedmanagement and the reorientation of UNDP tomake it more of a learning organization. Theoverall improvement in project performancereflects some of these changes. Their full effect,however, can only be understood in a broaderperspective by uncovering the contributions ofthese changes to the many challenges experi-enced at the programme and project levels.
The comprehensive framework provided by theshift to RBM and the quantitative analysis pre-sented above have highlighted a number ofoverarching themes that play a vital role indetermining development effectiveness. At theinstitutional level, important factors includestrategic focus, comparative advantage andpartnership. At the operational level, owner-ship, capacity building and sustainability are keyconcerns. These themes are assessed in greaterdepth in the following chapter, drawing onqualitative evidence from evaluations.
0 %
10 %
20 %
30 %
40 %
50 %
60 %
70 %
80 %
90 %
100 %
Level of National Ownership
Satisfactory PoorSignificant
V. Good Good Satisfactory Poor
perc
enta
ge o
f im
med
iate
obj
ectiv
es a
chie
ved
FIGURE 4: ACHIEVEMENT OF IMMEDIATE OBJECTIVES
BY LEVEL OF NATIONAL OWNERSHIP (ALL PROJECTS)
II.
UN
CO
VE
RIN
G U
ND
P D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
EV
ALU
ATI
VE
FIN
DIN
GS
14
Evaluating projects, programmes and the roleof UNDP in promoting development effective-ness requires a two-pronged approach. At theinstitutional level, the goal is to determine theimpact of the development strategy of UNDP,which has increasingly emphasised a shifttowards upstream interventions. Evaluative evi-dence shows that as a corollary to this strategicfocus there is a growing recognition that devel-opment effectiveness hinges on the organiza-tion's ability to identify its comparative advan-tages, build on them and engage in partnershipswith others to offer coherent and efficientstrategies. At the operational level, deliveringon ownership, capacity building and sustain-ability requires a thorough understanding of therole of UNDP, its performance and the chal-lenges it faces. Sustainability has traditionallybeen understood from a project perspective, andwill require re-examination in the context of theincreasing focus on promoting changes at thepolicy level.
The findings examined below are based onevaluative evidence drawn for the most partfrom the ROAR, the UNDP Essentials publica-tion and the strategic and thematic evaluationsconducted by the UNDP Evaluation Office.
Strategic Focus and ComparativeAdvantage
The ROAR confirms that UNDP is workingincreasingly at the policy or upstream level andis generally performing well at that level.Nonetheless, a substantial share of UNDPresults occurs at the downstream level, involvingmany small-scale interventions. For example,
downstream activities outweigh upstream sup-port in the area of poverty eradication by two toone. While the downstream interventions havevalue, the ROAR analysis points to the need forstronger links between UNDP’s policy workand its direct interventions. Most micro inter-ventions are focused at the community level,especially in Africa, although in a few countriesexperience gained at the downstream level isbeing fed into the design of policy and institu-tional frameworks – by broadening the access ofthe poor to micro-finance, for example.
The findings from the ROAR are supported bysubstantive evidence drawn from strategic andthematic evaluations. In the area of decentrali-sation, an evaluation conducted by theEvaluation Office in collaboration with theGerman Federal Ministry for EconomicCooperation and Development (BMZ) sup-ports UNDP putting greater emphasis onupstream policy advice. The report highlightsmany areas of strategic importance, stressingthe need for strategic management approaches,developing multi-dimensional capacities,emphasising the principle of subsidiarity andpromoting greater monitoring, risk-taking andrisk management. At the same time, it agreesthat UNDP should continue to support, in aselective fashion, field-level operational projectsso as to provide an empirical basis for advocat-ing innovative measures.
With respect to policy work supported byUNDP, not all outcomes have the same devel-opment value. Outcomes dealing with advocacyand awareness building can be seen as initialoutcomes, while those relating to the actual for-
II. Uncovering UNDP Development Effectiveness: Evaluative Findings
II.
UN
CO
VE
RIN
G U
ND
P D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
EV
ALU
ATI
VE
FIN
DIN
GS
15mulation and implementation of policies are moredirectly linked to actual development change.
UNDP appears to do well across all regions inadvocacy and in promoting awareness of sus-tainable human development issues and poli-cies, with national human development reports(NHDRs) as a key instrument. In terms of theformulation and implementation of policies,there is greater differentiation across theregions. In Africa, for instance, UNDP is concen-trating on pro-poor policy formulation, while inEurope and the Commonwealth of IndependentStates it is focused mainly on promotingawareness of sustainable human development.
However, in devising strategic policy interven-tions, findings from the ROAR stress the need toidentify potential linkages across sectors andthemes. One area of concern has been the lack ofsynergies between programmes in the fields ofpoverty alleviation, good governance and gender.
The recent emphasis on maximising the use ofscarce resources has highlighted the need fordevelopment institutions, and UNDP in partic-ular, to identify the areas in which they enjoycomparative advantages and thus need to focustheir efforts. The ROAR stresses the need toassess the comparative advantages of UNDP ona case-by-case basis. In this regard, thematicevaluations provide valuable insights into thepotential that may be unlocked by the strategicmanagement of these comparative advantages.
Post-conflict
The Evaluation Office undertook a strategicevaluation of the work of UNDP in post-con-flict situations in late 1998, focusing on reinte-gration programmes. The evaluation teamfound that UNDP, along with other develop-ment actors, has faced difficulties in shaping itsresponse to post-conflict situations over the lastdecade. Despite this, the majority of UNDPstakeholders and partners, especially pro-gramme country governments, stressed theunique strengths of the organization – such asits constant field presence, its neutrality and itscross-sectoral mandate to address the needs ofthe entire community rather than of one target-ed group – as vital assets during complex andpolitically charged situations. The team foundthat UNDP has contributed significantlythrough area-based reintegration programmesthat target the entire community and not justone target group, most notably in CentralAmerica and the horn of Africa, and has pro-vided valuable technical assistance to nationalde-mining efforts, especially in Cambodia.
Decentralisation
Starting from the premise that decentralisationefforts need to build on the comparative advan-tages of each institution involved, evaluation ofUNDP’s decentralisation projects has shownthat UNDP enjoys significant advantages vis-à-vis other donors in addressing issues of decen-tralisation and governance. In particular, itscross-sectoral mandate and its recognised neu-trality, independence and objectivity renderUNDP well equipped to overcome the politicalnature of decentralisation. In many instances,interviewees appreciated the fact that UNDP isnot pursuing any political or economic agenda.Furthermore, its well-established country pres-ence was perceived as important in understand-ing local particularities and supporting long-term initiatives. In addition, the value of
Poverty Alleviation
Gender Good Governance
II.
UN
CO
VE
RIN
G U
ND
P D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
EV
ALU
ATI
VE
FIN
DIN
GS
16UNDP’s use of innovative approaches such asparticipatory processes was recognised. Identifyingone’s strengths has consequences at the institu-tional and organizational level. The evaluationconcluded that, based on the recognition of itsposition of considerable advantage, UNDPshould change its image from that of simply afunder to one of an active catalyst of change.
Partnership
The focus on each institution’s comparativeadvantage in turn reinforces the value of part-nerships and aid coordination in order to guar-antee balance and coherence in developmentinterventions. Furthermore, RBM has shiftedthe focus of the organization from outputs –which could feasibly be produced by one organ-ization – to outcomes – which necessarilyrequire the efforts of many groups workingtogether. Placing the emphasis on outcomes hastherefore strengthened commitment to ensurethat UNDP fully invests in partnerships.Hence, the largest gains in effectiveness arelikely to lie in nurturing strong partnerships aspart of outcome-oriented coalitions, nationaland international, in support of specific devel-opment change.
In this context, UNDP and the World Bankundertook parallel evaluations in 1999 of theirrespective coordination mechanisms, theRound Table Mechanism (RTM) and the
Consultative Group Meetings. The EO pro-vided close technical assistance to theManagement Development and GovernanceD i v i s i o n ( M D G D ) o f t h e B u re a u f o rDevelopment Policy, which managed the evalu-ation. UNDP and the World Bank workedseparately and produced two reports for presen-tation to the OECD/DAC.
The evaluation of the RTM found that themechanisms have been relevant to the contextsin which they have worked, providing key link-ages to the donor community for the pro-gramme country. The RTM has also con-tributed to facilitating and focusing countries’efforts in defining their policies and pro-grammes, despite the fact that its follow-upmechanism remains relatively weak. The mech-anism has also allowed for the mobilisation ofsignificant financial resources. However, fur-ther information is needed as to the additional-ity of donor pledges and the value of the RTMas a vehicle for mobilising these resources.
To follow up on the findings and recommenda-tions of the two reports, the OECD/DAC, theWorld Bank and UNDP jointly organized aDevelopment Partnership Forum in December1999. Forum participants reviewed their experi-ences with aid-coordination processes in lightof the two evaluations’ findings and recommen-dations. To build on the momentum of theForum, each of the co-sponsoring organizationshas agreed to strengthen their cooperation.
BOX 3: ROLES OF PARTNERSHIP
• Collaboration at the operational level: the need to focus on one’s comparative advantage highlights the importance of joiningforces and combining expertise in the field.
• Coordination in knowledge sharing: consolidating evidence in the field and disseminating the findings among dfevelopmentinstitutions enhances effectiveness by enabling learning from others.
• Synergies in capacity building: at the UNDP-sponsored Beijing Conference on evaluatiion capacity participants agreed on thenecessity to design a common framework so that evaluation efforts can build on synergies and gain from broad knowledge sharing.
II.
UN
CO
VE
RIN
G U
ND
P D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
EV
ALU
ATI
VE
FIN
DIN
GS
17Cooperation and partnership among donorinstitutions are required at all levels and acrossall types of interventions (see Box 3). Frompost-conflict relief to institution building, thefocus on comparative advantage highlights theneed to establish efficient coordination at theproject and programme levels, where agenciesidentify their respective areas of greater expert-ise and work in synergy with one another. In thearea of decentralisation, evaluative findingshave emphasised that success hinges upon theability to promote stronger operational partner-ships and coordination among donors to tacklethe highly cross-sectoral nature of decentralisa-tion and governance programmes.
Ownership
The emphasis on aid coordination is also inex-tricably linked to renewed efforts to strengthenthe principle of country ownership. In additionto its commitment to establish fruitful partner-ships at the Development Partnership Forum,UNDP has pledged to promote country owner-ship in development cooperation and coordina-tion, and to work with its multilateral and bilat-eral partners to strengthen national capacities tothat end. The World Bank has also designatedownership as an essential component of sus-tainable reform.
As previously highlighted, evaluation evidenceshows that the level of country ownership ispositively correlated to the performance of theproject. The mode of execution of UNDP-sup-ported interventions is itself associated withvarying levels of ownership. Since 1992, thoseinterventions implemented through NationalExecution (NEX) appear to have been linkedwith higher levels of country ownership, andcorrespondingly improved results. This findingmust, however, also be seen in the broader con-text of the need for the simplification andimprovement of the NEX modality, highlight-
ed in an evaluation of NEX in 1995. Not sur-prisingly, NEX has proved less effective wherenational government is weak or non-existent.
According to the portfolio of projects availablein the evaluation database, NEX projects enjoyhigher levels of government support, as judgedby the evaluators, with 34 per cent of all NEXprojects receiving very good support, 35 percent receiving good support, 19 per cent receiv-ing satisfactory support and 12 per cent receiv-ing poor support. Non-NEX projects, on theother hand, had lower levels of support, with 17per cent receiving very good support, 25 percent receiving good support, 43 per cent receiv-ing satisfactory support and 15 per cent receiv-ing poor support.
In examining NEX versus non-NEX perform-ance, we see that overall the two are not per-forming that differently. With regard to NEXprojects, 83 per cent were rated at or above sat-isfactory in the period 1987-1991 and 86 percent had such ratings in the period 1992-1998.At the same time, 77 per cent and 83 per centof non-NEX projects were rated at or abovesatisfactory in the respective time periods.However, this overall similarity masks two facts:a larger percentage of NEX projects is ratedvery good; and an examination of the break-down between satisfactory, good and very goodprojects shows that NEX is significantly con-tributing to the upward trend in improvingproject design.
Interestingly, NEX projects are assessed to havea higher quality of management than non-NEXprojects. This is one way in which NEX appearsto contribute to an improved ability of UNDP-supported interventions to achieve their overallobjectives. As shown in Figure 5, page 18, NEXprojects are rated higher for quality of manage-ment, especially at the key levels of very goodand good management.
II.
UN
CO
VE
RIN
G U
ND
P D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
EV
ALU
ATI
VE
FIN
DIN
GS
18In terms of achievement of outputs and imme-diate objectives, NEX is associated with animprovement both by lowering the percentageof projects performing poorly and by increasingthe share that performed particularly well.
From 1992 to 1998, the proportion of NEXprojects in the total pool of evaluations waslarge. Therefore, NEX project performancerates clearly contributed to the improved over-all rates for UNDP in the most recent period(Figures 6 and 7).
According to a 1995 evaluation, NEX projectsplay a significant and positive role in promotingan increase in ownership, self-reliance, capacitybuilding and sustainability. Quantitative evi-dence does not, however, diminish the need touncover a number of issues and challenges facedin the implementation of NEX projects. In par-ticular, it is important to understand underwhat circumstances NEX is best able to con-tribute to improved development effectiveness.It is also vital to address the organizationalchallenges that NEX raises. In a recent evalua-tion commissioned by the Evaluation Office onthe relationship between UNDP and UNOPS,concerns were raised over the limited scope ofintervention afforded to UNOPS in NEX proj-ects. A UNDP evaluation of the programmeapproach, completed in 1998, concluded thatthe programme approach, which commenced atbroadly the same time as NEX, had also beensignificant in enhancing ownership. The UnitedNations agencies have defined the programmeapproach as a process that helps governmentsto formulate national priority developmentobjectives, and to realise them through corre-sponding national programmes formulatedand implemented in a coordinated andparticipatory manner.
0
5
10
15
20
25
30
35
40
45
V. Good Good Satisfactory
non-NEX NEX
Poor
Quality of Management
FIGURE 5: QUALITY OF PROJECT MANAGEMENT
FOR NEX AND NON-NEX PROJECTS (1992-1998)
0 %
10 %
20 %
30 %
40 %
50 %
60 %
Significant Satisfactory Poor
Non-NEX NEX
FIGURE 6: ACHIEVEMENT OF OUTPUTS
FOR NEX AND NON-NEX PROJECTS (1992-1998)
0 %
10 %
20 %
30 %
40 %
50 %
60 %
70 %
Significant Satisfactory Poor
Non-NEX NEX
FIGURE 7: ACHIEVEMENT OF IMMEDIATE OBJECTIVES
FOR NEX AND NON-NEX PROJECTS (1992-1998)
II.
UN
CO
VE
RIN
G U
ND
P D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
EV
ALU
ATI
VE
FIN
DIN
GS
19More generally, the issue of ownership can gen-erate both synergies and tensions with otherfactors that are part of development effective-ness. A World Bank study highlights the criti-cal tension between ownership and partnershipas governments and external donors may havedifferent agendas.
Sustainability
Evaluation data suggest a significant increase inthe sustainability of UNDP-supported inter-ventions over the past decade. The share ofinterventions that were rated as having a goodor high level of sustainability rose from onethird of projects approved during the 1987-1991 period, to two thirds for projects approvedduring 1992-1998. Figure 8 below indicates that,as with other components of development effec-tiveness, projects implemented through the NEXmodality are rated as having higher expectedlevels of sustainability than non-NEX projects.
More generally, the combination of partnership,ownership and capacity building, while carryingpotential for tension, lies at the heart of efforts
to promote sustainability. A study of the micro-finance area in particular provides an importantlesson: promoting the sustainability of micro-finance institutions and serving the needs of thepoor do not have to be conflicting goals. Astudy of micro-finance institutions by theConsultative Group to Assist the Poorest(CGAP) found no clear trade-off betweenreaching the poor and achieving financial via-bility. A number of micro-finance institutionsare cited that have achieved sustainability whilecontinuing to serve very poor clients and offerloans of small size.
Capacity Building
Engaging local governments and putting themin the driver’s seat of the development processrequires that efforts be made and resultsachieved in building local capacities to under-take sound political, economic and social poli-cies. As underlined by the World Bank, "insti-tutional capacity to manage and co-ordinate aidis often a missing condition for country-ledpartnership." Achieving ownership and build-ing capacity go hand-in-hand and should beapproached simultaneously as mutually rein-forcing processes.
Demand for strengthening the capacity of keygovernance institutions is high. Institutionbuilding now accounts for 38 per cent ofUNDP core funding resources and 58 per centof its non-core resources. The term embracesmany activities, from establishing transparent
0 %
10 %
20 %
30 %
40 %
50 %
60 %
70 %
80 %
No Partially Yes
Non-NEX NEX
FIGURE 8: SUSTAINABILITY
FOR NEX AND NON-NEX PROJECTS (1992-1998)
Ownership
Capacity Building
II.
UN
CO
VE
RIN
G U
ND
P D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
EV
ALU
ATI
VE
FIN
DIN
GS
20legal framework to developing local entrepre-neurship. Examples drawn from evaluative evi-dence are provided for each of these below.
Assisting Legislatures
Legislative activities have a direct impact on theenabling environment for sustainable humandevelopment. Building legislatures can promotedemocracy, strengthen the rule of law, supporthuman rights and increase efficiency in themarketplace. Development agencies have pro-vided legislative assistance in many forms,including building mechanisms for enhancingthe relations between the legislators and theconstituents and supporting external partnerswho can influence and monitor them, such asnon-governmental organizations, citizengroups, the media, other governmental branch-es and political parties.
Evidence gathered in the UNDP Essentialspublication (see Box 8) provides a number ofkey lessons from experience in assisting legisla-tures. One lesson is the need for donors tounderstand and anticipate socio-economicchanges in the country. This has required thedirect and personal involvement of the head ofthe donor mission and the strengthening of thedonor’s ability to analyse and understand polit-ical dimensions. For example, after the militarygovernment in Gambia declared an interest inmoving towards democracy, UNDP initiated anumber of workshops in key areas that identi-fied the parliamentary needs. Based on thesediscussions, the technical assistance strategywas designed and included in the country coop-eration framework.
Another important lesson is the need to usenon-partisan approaches and personnel in whatare sure to be politically sensitive situations andprogrammes. Successful programmes haverelied on steering committees or advisory
boards in which all blocks of government wererepresented. Other successful approaches haveincluded the use of existing legislative commit-tees that are composed of members of differentpolitical parties, or ensuring that all trainingand equipment are distributed fairly among dif-ferent parties. One UNDP training programmein Mozambique diffused political tensions byabandoning the use of all political party namesduring the training and calling the groups bygeneric names instead. Attention to tensionsand details can be critical in ensuring that assis-tance is successful.
Entrepreneurship Development
The primary obstacle to sound economicgrowth in a country is often a scarcity ofdynamic entrepreneurs rather than a lack ofcapital, labour or land. Dynamic entrepreneursbring these other factors together to creategrowth. Developing entrepreneurship, then, isan effective way of reducing poverty and target-ing economic growth. Entrepreneurship devel-opment (ED) consists of enhancing entrepre-neurial skills and knowledge through structuredtraining and institution building. ED aims toenlarge the base of entrepreneurs so as toincrease the rate at which new ventures are cre-ated and thereby accelerate employment gener-ation and economic growth.
The first lesson learned by the findings pub-lished in Essentials is the importance of havinga clear understanding of the feasibility and theobjectives of any programme for ED. OftenED is confused with promoting self-employ-ment. These two concepts are in fact quite dif-ferent, require different approaches and shouldbe carefully distinguished from one another inthe design of an ED programme. ED should beabout helping to build dynamic businesses andadding value to a growing sector. Supportingentrepreneurs therefore often requires a more
II.
UN
CO
VE
RIN
G U
ND
P D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
EV
ALU
ATI
VE
FIN
DIN
GS
21dynamic and growth-oriented view of businessdevelopment. In Kenya for example, an evalua-tion of an ED programme recommended that itshould be oriented towards building partnershipsand small firms and enhancing their outreach,not just focus on promoting self-employment.
Another important lesson is the need to identi-fy a market that has room for innovation andgrowth potential. Successful entrepreneurialbusinesses are more often those that meet a pre-viously neglected or unforeseen need or presenta new and better way of doing something. Assuch, evaluators have found that good ED pro-grammes must help entrepreneurs to recogniseand respond to unique opportunities, or todiversify based on knowledge of a particularsector and innovate. For example, in Nepal awomen’s ED programme found that manyentrepreneurs had little business experience andwere likely to copy successful business modelsrather than to innovate. The programme musttherefore either help them to ensure that themarket has room for the same types of productsor services or assist them to diversify.
Building Civil Society
Country ownership is but the first step in pro-moting effective participation by local stake-holders. Development effectiveness requiresdevolution of power and resources to the locallevel and reaching out to all stakeholders andbeneficiaries. For example, in discussing formu-lation of national policy and the creation of anenabling environment for SMEs, the UNDPEssentials publication cites evidence that partic-ipation by SMEs in decision-making is veryimportant. In the past policies were often draft-ed without consulting SMEs. As a result, theytended to be largely irrelevant to their needs,failing to target weaknesses or build onstrengths. Under the section on programmedesign and implementation, Essentials goes on
to highlight the importance of involving theprivate sector in the design and implementationof any programme targeting SMEs. Evaluatorsespecially stress the use of United NationsVolunteers to garner the expertise of nationalprivate sector actors.
While the ROAR shows significant evidence ofUNDP promoting civil society and public-pri-vate actor linkages, evaluations of projects andprogrammes on decentralisation highlight theneed to ensure broad-based participation as acentral feature of effectiveness. Often, theinclusion of actors from civil society and theprivate sector remains an aspiration rather thana reality. Evaluative findings also point tothe lack of real fiscal decentralisation, whichhampers the true devolution of power to thelocal level.
Developing Evaluation Capacity
Another integral part of responsible civil socie-ty development, and the enhancement ofnational capacity in general, is the developmentof evaluation capacity (ECD) at the local level.Evaluation is recognised as a mechanism forpromoting accountability for the use of publicresources as well as an instrument for organiza-tional learning and institution building. Thenexus between ECD and good governance can-not be ignored. Development agencies, workingwith national partners, have the potential tocontribute to development effectiveness bystrengthening national capabilities in thisregard. This lies at the core of the promotion byUNDP of ECD, defined as the ability of publicinstitutions to manage information, assess pro-gramme and project performance and respondflexibly to new demands. In this context, ECDis a necessary component in the UNDP pro-gramme of assistance.
II.
UN
CO
VE
RIN
G U
ND
P D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
EV
ALU
ATI
VE
FIN
DIN
GS
22The findings from the ECD Beijing conferenceheld in October 1999 are highly instructive inthis regard. In addition to citing systemic andmethodological obstacles, many participantsstressed the challenge of introducing a cultureof evaluation at the national level and guaran-teeing the independence of the evaluation
function. ECD hence requires a collaborativeeffort structured around political commit-ment, institutional support, local training andeffective partnerships. Efforts to promoteECD have already yielded highly conclusiveresults (see Box 4).
BOX 4. SRI LANKA: MOVING FROM CONCEPTS TO PRACTICE
The Area-Based Growth with Equity Programme funded by UNDP and the Government of Sri Lanka provides an interesting exampleof the application of results-based monitoring and evaluation and the role UNDP plays in ECD. It involves:
• Introduction of results-based monitoring and evaluation at the national, provincial and district levels, with subsequent extensionto divisional levels.
• Preparation and translation into two national languages (Sinhala and Tamil) of training materials.
• Setting up, with assistance from UNICEF, of the Sri Lanka Evaluation Association under the monitoring and evaluation componentof the Area-Based Growth with Equity Programme. With the establishment of the Evaluation Association, evaluation in Sri Lankais now being extended to civil society as well. The Association meets regularly and has made contacts with other nationalevaluation bodies overseas and has organized several training programmes for its members.
• Support for a Network of Social Mobilization Practitioners to document the current best practices of participatory monitoring andevaluation programmes in the country.
III.
UN
LOC
KIN
G D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
CU
RR
EN
T M
EC
HA
NIS
MS
AN
D D
IRE
CTI
ON
S F
OR
TH
E F
UTU
RE
23
Unlocking development effectiveness is amulti-faceted endeavour. It calls for action atboth the institutional level and in the imple-mentation phase, from strengthening a results-based culture within the organization to pro-moting deeper partnerships with other institu-tions.This section outlines what UNDP is cur-rently doing to provide these responses andthen indicates what remains to be done, basedon recommendations drawn from evaluativefindings.
Institutionalising Results-basedManagement
UNDP has sought to learn from the experi-ences of other agencies in developing andimplementing measures designed to strengthendevelopment effectiveness. In 1997, the UNDPEvaluation Office and the Swedish InternationalDevelopment Agency (Sida) produced a jointreview of performance measurement systems ofselected public sector and aid agencies. Thisserved as an initial building block for the intro-duction of RBM within UNDP. In October
1998, UNDP, Sida and the Aid EvaluationWorking Group of the OECD DevelopmentAssistance Committee organized a workshop inNew York to review the experience of bilateraland multilateral agencies in introducing RBMprinciples and practices. These consultationsand exchanges of experience resulted in severalkey lessons for strengthening the institutionali-sation of UNDP’s RBM system (see Box 5). Anattempt was made to ensure that each of theselessons was consciously reflected in the system.
A further general lesson that emerged was theimportance of stressing management overmeasurement. The fundamental goal of RBMis to improve development effectiveness, whichrequires helping managers to better manage. Incomparing RBM systems, the distinction issometimes made between managing by resultsand managing for results. The former is princi-pally oriented towards accountability and exter-nal reporting; the latter focuses on a cycle ofplanning, periodic performance and organiza-tional learning. In implementing RBM, UNDPmade a deliberate decision to emphasise man-agement and learning. This was based on an
III. Unlocking Development Effectiveness: Current Mechanisms and Directions for the Future
BOX 5. INSTITUTIONALISING RBM: LESSONS LEARNED
• The organization needs to set clear objectives for RBM itself: should RBM be a reporting and classification system (results measurement) or a strategic planning and management tool that can help improve organizational performance (resultsmanagement)?
• Any system must fit UNDP’s specific needs and culture. As an approach, RBM requires that the culture and specific nature ofthe organization be carefully taken into account. In particular, the unique mandate and decentralised structure of UNDP requireflexibility at the country level.
• Implementing RBM is a learning process. RBM depends on many aspects of organizational culture, policy and operational practice. Any system needs to be seen as a work in progress, evolving over a considerable period of time and incorporating flexibility to make changes as experiences are gained.
• It is essential to keep the approach simple. RBM should not lead to an increased workload. Instruments must be easy to under-stand, and their number limited.
III.
UN
LOC
KIN
G D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
CU
RR
EN
T M
EC
HA
NIS
MS
AN
D D
IRE
CTI
ON
S F
OR
TH
E F
UTU
RE
24unequivocal message from donors and staff:RBM must explicitly aim at changing the waythe organization is managed, fostering a strate-gic orientation and culture of performance.Improved external reporting was approached asvery important, but a secondary benefit.
At headquarters, Strategic Results Frameworks(SRFs) are intended to improve UNDP sub-stantive accountability to national stakehold-ers and the Executive Board. They will assistmanagers in judging whether the overall resultsof UNDP assistance worldwide meet the goals,principles and standards set forth in theMission Statement, in Executive Board deci-sions and in operational and thematic policies.In addition, for the first time, the SRFs will laythe basis for a funding strategy to supportapproved programmes whose results are clearlyidentified and monitored.
In support of the above, UNDP has instituteda more formal results-based accountability rela-tionship with its resident representatives. A"compact" identifies the key strategic (pro-gramme and management) results for which aresident representative is held accountable overa given period. The results of the compact arederived from the larger set of planned resultscontained in the SRF. Items chosen for thecompact reflect conscious decisions by the man-agers concerned regarding the most importantresults for which they are to be directly heldaccountable.
Although the problems associated with measur-ing and attributing development outcomes arewell known, the drive to improve developmenteffectiveness was felt to require that UNDPresults-based management ground its conceptof results at the outcome rather than the out-put level. While both levels of accomplishmentare important and are reflected in the system,this decision to emphasise outcomes was based
on two considerations. The first was the crisis ofconfidence facing the development communityas many question the effects of investments indevelopment. In view of this crisis, it was feltnecessary for UNDP to portray a significantand honest picture of the consequences of itsinterventions.
Secondly, since many of UNDP’s most impor-tant contributions result from what some havecalled "soft interventions" – advocacy, policydialogue and institutional strengthening, fieldpresence – it was deemed essential to projectand capture outcomes of these particularlyhard-to-measure and hard-to-attribute areas.While this effort continues to be a work inprogress, UNDP has committed itself to theproposition that "soft interventions need tohave hard outcomes," and to incorporating suchinterventions into its system of results-basedmanagement.
Strengthening SubstantiveAccountability
Currently, no comprehensive framework existswithin UNDP for ensuring substantiveaccountability. As discussed above, the ROARand the SRF go some way towards filling thisgap, given their emphasis on objective perform-ance assessment through the use of indicators.
The systematic monitoring of evaluation com-pliance provides a further element of substan-tive accountability. The number of evaluationreports received by the UNDP EvaluationOffice increased substantially in 1998 and 1999compared with the 1996-1997 period. Overall,a 71 per cent rate of strict compliance wasattained for projects originally approved in1990, exceeding the target of 70 per cent.
III.
UN
LOC
KIN
G D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
CU
RR
EN
T M
EC
HA
NIS
MS
AN
D D
IRE
CTI
ON
S F
OR
TH
E F
UTU
RE
25The contribution made by the evaluation com-pliance instrument is modest, however, and inrecognition of this an attempt has been made toprovide additional mechanisms for substantiveaccountability. Some progress has beenachieved through the introduction of evaluationplans in 1998, although these have yet toachieve their full potential.
Evaluation plans are designed to provide rele-vant advance information on the areas in needof assessment. While traditional compliancereporting involves a long time lag – often six orseven years – in its assessment of performance,evaluation plans furnish a more timely instru-ment for determining country-level substantiveaccountability. Evaluation plans could also pro-vide a tracking framework to link recommenda-tions to decision-making processes and imple-mentation. They were made a requirement in1998 and were included in the UNDPProgramming Manual in April 1999.
Use of the evaluation plans makes it possible tocalculate the proportion of mandatory evalua-tions to be conducted, broken down by region-al bureau. While initial performance figures for1999-2001 will be presented next year, prelimi-nary estimates, by area of focus, of the evalua-tions planned in 1999 show differences inemphasis across bureaux. For example, povertyis the largest area for Africa, accounting for 41per cent of planned evaluations, followed byenvironment. Governance ranks highest forArab States, at 50 per cent, and for both LatinAmerica and the Caribbean and Asia and thePacific, at over one third of planned evaluations.
The assessment of country office evaluationplans, combined with strong encouragement tocountry offices to submit plans, represents animportant step forward. The plans provide animportant real-time mechanism for improvedaccountability by ensuring that the portfolio of
evaluations that a country office intends tocarry out is consistent with UNDP prioritiesand is likely to allow lessons to be drawn forfuture programming. Evaluation plans offerUNDP an avenue through which it willincreasingly be possible to offer country officespro-active guidance on the types of evaluationthat are likely to be most suitable, and to drawattention to the existence of similar evaluationsconducted by others. The recently establishedEvaluation Plan Database, now accessiblethrough the Evaluation Office Intranet site,allows country offices and other units managingevaluations to prepare their evaluation planson-line, and makes them available more broad-ly. This will allow users to identify possible areasof collaboration in organizing evaluationsaddressing similar issues, as well as to identifypotential sources of lessons themselves.
Methodologies for Assessing Impactand Improving Feedback
The ROAR provides a systematic assessment ofprogress made by UNDP vis-à-vis its strategicobjectives. This in turn requires the develop-ment of complementary tools for impact assess-ment, an exercise that examines to what extentprogress has been translated into concretedevelopmental changes for the poor. Inresponse to this chal lenge, in 1999 theEvaluation Office developed a methodology forcountry-level impact assessments (CLIAs).These aim to: (a) document all UNDP inter-ventions, including policy dialogue and advoca-cy; and (b) identify all factors that have led tothe impact in order to draw appropriate lessons.
Since October 1999, the Evaluation Office hasbeen testing its CLIA methodology in BurkinaFaso. Further testing will be conducted inMalawi and in an Asian country. Once this hasbeen completed and the methodology has been
III.
UN
LOC
KIN
G D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
CU
RR
EN
T M
EC
HA
NIS
MS
AN
D D
IRE
CTI
ON
S F
OR
TH
E F
UTU
RE
26appropriately refined, the Office will conductthree to five assessments a year. The focus onimpact and the attention to building in-housecapacities to evaluate impact will contribute tothe wider promotion of results-based approaches.
RBM requires the rapid provision of feedbackto allow for corrective steps in policy or imple-mentation. An instrument seeking to respondto this need is the Evaluative ResearchProgramme (ERP), developed by theEvaluat ion Off ice in 1999. The ERP isdesigned to meet the need of UNDP for "justenough knowledge, just in time," and thus pro-vide relevant evaluative knowledge as and whenit is needed. It represents an experimentalapproach to the challenge of making evaluationmore relevant to the organization and will pro-vide lessons for policy and strategy formulationmore rapidly than is normally feasible. The pro-gramme of work was approved in early 2000 onthe basis of the ROAR findings, the focuses ongovernance-poverty linkages and linkagesbetween macro policies and micro-level initia-tives.
Linking Evaluation Recommendationsto Decision-making
Better knowledge of the UNDP contribution todevelopment changes is a prerequisite forachieving long-term improvements in its effec-tiveness. However the ability to translateknowledge into change is also necessary. Inother words, the policy-making process must beresponsive to the findings emerging fromresults-based systems. Substantive accountabil-ity, therefore, requires efficacious mechanismsfor ensuring effective follow-up to the recom-mendations of evaluations, as well as trackingprogress in implementation. Action is requiredwith respect to both country-level evaluationsand corporate-level strategic and thematic eval-
uations. The following analysis highlights sig-nificant limitations in current mechanisms.
Country Level Follow-up
The annual project/programme report (APR)provides, in principle, a means of tracking theimplementation of recommendations containedin country-level evaluations. While the APR isan essential part of the project cycle, it is clear-ly not working as it should, nor is it facilitatingthe mainstreaming of RBM. Concern overthese shortcomings resulted in the decision toconduct a survey of various Evaluation Networkmembers in country offices. Its purpose was toassess the extent to which recommendationscontained in country-level evaluations are beingimplemented, and to identify what the obstaclesto effective follow-up might be. The surveyrevealed that there is no systematic mechanismto ensure that recommendations are acted upon.While several country offices had developedtheir own semi-formal approaches, includingthe regular preparation and monitoring ofimplementation blueprints that include anaction plan linked to recommendations con-tained in each evaluation, most respondentsfavoured a more systematic approach.
The survey highlighted several factors influenc-ing the likelihood of effective follow-up of rec-ommendations (see Box 6).
Furthermore, it was felt that ownershiprequired the full participation of all majorstakeholders, and that if this conflicted withconsiderations of objectivity, areas of contention
BOX 6. EFFECTIVE FOLLOW-UP OF RECOMMENDATIONS
Essential components include:
• the quality of the evaluation itself;
• the relevance and realism of the recommendations,
• the degree of ownership felt by the key stakeholders.
III.
UN
LOC
KIN
G D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
CU
RR
EN
T M
EC
HA
NIS
MS
AN
D D
IRE
CTI
ON
S F
OR
TH
E F
UTU
RE
27should be addressed openly rather than by cur-tailing participation.
The majority of respondents underlined thevalue of mid-term evaluations over terminalevaluations. This finding is consistent with theincreased share of the total that mid-term eval-uations represent, up from 23 per cent for 1997-1998 to 37 per cent in 1999. It is also consistentwith the emphasis the Evaluation Office hasplaced on the need for evaluation to providemanagers with relevant information when theyrequire it. While mid-term evaluations are seenas highly relevant, with the potential to assistmid-course corrections, they are often not actedupon. A possible function of the evaluationplans could be to provide a system of accounta-bility that tracks whether recommendations arebeing heeded. Terminal evaluations remainimportant as a source of generic lessons and foroverall accountability.
Corporate-level Follow-up
The 1999-2000 programme of strategic andthematic evaluations managed by theEvaluation Office corresponds closely to thestrategic priorities of UNDP as revealed both inthe MYFF and the Business Plans. It has beendeveloped through dialogue with UNDP seniormanagers. The mechanism to ensure that eval-uation recommendations are fed into decision-making includes the preparation of action briefs
by the Evaluation Office, summarising themain findings and recommendations of eachevaluation as well as their corporate implica-tions. Action briefs are submitted to theExecutive Team for appropriate decision. TheEvaluation Office keeps a record of the discus-sions and decisions made in this forum to facil-itate the monitoring of implementation. Forevaluations presented to the Executive Board,formal management responses are now preparedand submitted with the evaluation reports.
Considerable attention is being given to followthrough of the recommendations included inthe Strategic and Thematic Evaluations. Withrespect to the crisis and post-conflict evalua-tion, the report’s findings and recommenda-tions were presented to the Administrator ’sTransition Team and reflected in their report inOctober 1999, as well as in the Crisis and Post-Conflict Business Plan, which was submitted tothe Executive Board in April 2000.
The decentralisation evaluation was activelyfollowed up on, with UNDP and BMZ co-sponsoring a seminar in Berlin in May 2000 topresent its main findings to development pro-fessionals and practitioners from developingcountries and donor institutions. The results ofthis seminar are being reviewed and will formthe basis for UNDP action jointly with otherpartners (see Box 7).
BOX 7: FOLLOWING THROUGH ON EVALUATION FINDINGS – DECENTRALISATION
Recommendations include:
• Feeding the results of the evaluation and the seminar into a high-level policy dialogue, inter alia through the OECD-DAC WorkingGroup on Aid Evaluation, to link the lessons learned to overall aid policy framework;
• Undertaking joint programmes and building effective partnerships. In particular, the joint evaluation and the Berlin seminar havesignificantly contributed to a stronger constituency for UNDP in Germany. Specifically, it is recommended that UNDP takeadvantage of this and seek the assistance of its German partners at BMZ in bringing key issues to the OECD-DAC forum;
• Instituting more systematic feedback and sharing of lessons learned among partners. It is critical that UNDP strengthen itscorporate knowledge management system to effectively link all country offices, enabling them to access lessons learned fromwithin and outside the organization.
III.
UN
LOC
KIN
G D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
CU
RR
EN
T M
EC
HA
NIS
MS
AN
D D
IRE
CTI
ON
S F
OR
TH
E F
UTU
RE
28op a culture of learning. This has included theestablishment of the Sub-regional ResourceFacilities (SURFs). These networks aredesigned to provide expert referrals for practi-tioners, technical and thematic backstoppingand networking for a better dissemination ofknowledge. They identify and disseminate bestpractices, link and assemble practitioners for asharing of knowledge, and make the dissemina-tion of such knowledge more effective throughthe use of information technology and theInternet. So far, according to a recent evalua-tion, the development of the SURFs has beenuneven, due in part to the absence of a coherentglobal strategic framework for knowledge man-agement at the senior level, and to a lack of ade-quate resources. Other findings and recommen-dations are presented in Annex II.
Evaluation Network
A further element designed to enhance devel-opment effectiveness through improved learn-ing and knowledge dissemination is theEvaluation Network (EVALNET). EVAL-NET creates links among practitioners whoshare expertise and knowledge in the area ofevaluations across countries and regions. Thisnetwork has promoted practical learning byallowing UNDP staff to participate in evalua-tions being carried out in other country offices,as well as at the global level. By targeting bothpractical learning and the sharing of this knowl-edge, EVALNET has allowed the learning andcapacity building for evaluations to take rootsubstantively throughout UNDP. The firstinternal review of this recently-developedinstrument points to the need to increaseopportunities for more regular direct partici-pation of EVALNET members in evaluationsoutside of their duty stations, in order to max-imise the potential for the cross-fertilisationof ideas.
Acquiring Knowledge and SharingLessons Learned: The Value ofPartnerships
The Administrator has placed knowledge andlearning in UNDP at the core of the newBusiness Plan. Generating lessons and bestpractices and improving the mechanisms bywhich this knowledge is disseminated areimportant means for enhancing developmenteffectiveness. The Administrator states in theBusiness Plan, "The UNDP organizationalmodel for the future should be based on dis-persed capabilities and decentralised networks.Above all, the basis for our knowledge andlearning – the bedrock of the intellectual capitalthat must become our trademark – should belocated closer to the field, where it can feed thewider network."
Learning is, by its very nature, a work inprogress. It has been targeted through numer-ous networks and products developed at head-quarters, and through organizational changes atvarious levels. Evaluative findings, in the area ofdecentralisation in particular, point to weak-nesses in the management and dissemination ofinformation and knowledge despite the recog-nised strength of UNDP’s research and devel-opment units. However, while the work isfar from completing the vision of theAdministrator, the groundwork for the futurehas been laid.
Sub-regional Resource Facilities
UNDP has worked to strengthen its abilities inthe area of knowledge and learning for a num-ber of years. In order to build a fully developedframework for knowledge management, UNDPhas centred its efforts on increasing access to,and sharing lessons and knowledge through,better networks and targeted products and serv-ices, thereby helping the organization to devel-
III.
UN
LOC
KIN
G D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
CU
RR
EN
T M
EC
HA
NIS
MS
AN
D D
IRE
CTI
ON
S F
OR
TH
E F
UTU
RE
29The programme of strategic and thematic eval-uations managed by the Evaluation Office dur-ing 1999 and the first six months of 2000 cor-responded closely to the corporate priorities ofUNDP, as reflected in both the Administrator’sBusiness Plans and the Results-orientedAnnual Report. Feedback from the UNDPExecutive Board suggests that the criteria usedto prioritise the agenda for strategic evaluationshave allowed UNDP management to effective-ly target issues relevant to current debate.
The major strategic and thematic evaluationsconducted during the 1999-2000 period havecovered subjects including post-conflict situa-tions, decentralisation and local governance, theUNDP/UNOPS relationship, SURFs and thecoordination conducted by the Bureau ofDevelopment Policy.
The acquisition of knowledge finds its true rel-evance within the framework of partnerships.Learning from others, and sharing lessonslearned from the organization’s own projectsand programmes through efficient networks,ensures in turn successful collaboration at theproject-implementation and policy-design lev-els. The value of partnership in tackling themany challenges associated with poverty allevi-ation is now firmly recognised across the devel-opment spectrum.
Essentials
The provision of concise information of direct,practical relevance to frontline country-officestaff and their counterparts is a major compo-nent of efforts to increase UNDP developmenteffectiveness (see Box 8). This is achieved withrespect to evaluation through the Essentialspublication. Essentials synthesises the main les-sons generated by evaluations within UNDPand in the field in general. Presented in a read-ily accessible and concise format, the publica-tion is a tool for development practitioners touse as a primer in a subject, and to guide themto more in-depth or specific information. Allissues of Essentials are available from theUNDP Evaluation Office or on the website athttp://www.undp.org/eo.
Thematic and Strategic Evaluations
Perspectives on development effectiveness andthe factors determining it are provided throughin-depth strategic and thematic evaluations.These tackle large policy issues through a thor-ough review of UNDP work, typically incorpo-rating an examination of a number of UNDPcountry programmes as well as a comprehensivereview of the policy framework. These evalua-tions may take up to a year or more to completeand are fed into the decision-making processthrough a series of meetings, briefs and presen-tations. They offer objective, independentinformation on topics of current import, asidentified by UNDP management.
BOX 8. PRACTITIONER-ORIENTED INFORMATION – ESSENTIALS
Essentials provides an opportunity for UNDP to scan a large number of current conceptual and evaluation documents and make keyfindings available to those who could benefit from them. The lessons it presents are culled from UNDP project evaluations carriedout in close collaboration with substantive units from the Bureau of Development Policy; UNDP policy or sector evaluations or eval-uations done by other organizations; conceptual documents on particular subjects; and discussions with practitioners and membersof organizations involved in the work. Four issues of Essentials have been published since November 1999, each one presenting les-sons learned, recommendations evaluators have made and country examples. Topics covered include Small and Medium EnterpriseDevelopment and Micro-finance. Issues covering each of UNDP’s major strategic areas are planned for the coming year.
III.
UN
LOC
KIN
G D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
CU
RR
EN
T M
EC
HA
NIS
MS
AN
D D
IRE
CTI
ON
S F
OR
TH
E F
UTU
RE
30concrete terms, and highlight relationshipsamong programme objectives, designs andresults. Four broad capacity building elementsare required in order to operationalise this func-tion in the context of results-based monitoringand evaluation:
• Provision of training or human resourcesdevelopment at all levels.
• Preparation for results tracking that includesindicator development, information gatheringand devising reporting formats and systems.
• Use of information and communicationtechnology.
• Building partnerships at various points of thedevelopment continuum.
This last element, partnership, is necessary inorder to reach a common understanding of thepurpose, benefits and use of results-based mon-itoring and evaluation, and to bring about acommon ownership of the process. The part-nerships should connect those who are respon-sible for results-based monitoring and evalua-tion at the centre with those who are responsi-ble for generating results. They also help toreduce misconceptions and fears with regard toresults-based monitoring and evaluation, as wellas to build understanding that such a system hasless to do with policing than with providing anaid and tool for enhancing development effec-tiveness.
2. Strengthening or development of an opera-tional environment within which results can beeffectively analysed and presented and lessonsfed into the policy development and implemen-tation processes. This relates to four broadcapacity building dimensions:
• The quality of the results-based monitoringand evaluation process and its outputs, which
Strengthening Evaluation CapacityDevelopment (ECD)3
The UNDP-sponsored Beijing Conferencehighlighted the main challenges facing ECDand laid the groundwork for future directions.In line with the overall RBM framework, eval-uation results need to be based on quantitative-ly measurable and/or qualitatively comprehen-sible parameters and indicators in order to bewidely accepted and used for assessing progress,bringing about changes in policies and prac-tices, etc. In other words, evaluations and theiroutputs have to be results based.
However, the central goal of ECD – to createstrong evaluation organizations at the nationaland local levels, capable of playing a significantrole in resource allocation, planning and policyformulation – still presents enormous chal-lenges. In most developing countries, especiallyin low-income countries, results-based moni-toring systems are still in early stages of devel-opment. In most cases regular monitoring isfocused on, or limited to, monitoring of finan-cial and physical performance and rarely, meas-uring outcomes and impacts of developmentprocesses. In such situations results-basedevaluations become indispensable to assess theeffects of the development processes being used.
Operationalisation of ECD can be conceptu-alised in terms of the following three dimensions:
1. Development of a conceptual understand-ing and an effective and reliable system –comprising software, hardware and humanresources. Such a system should capture data onthe effectiveness of development interventionsand associated trends, transmit information ontrends and effects of development initiatives in
3 The contributions of Dr. Adil Khan, Mr. Tilak Gunawardena, Ms. Solfrid Lien, and Ms. Rekha Thapa are gratefullyacknowledged.
III.
UN
LOC
KIN
G D
EV
ELO
PM
EN
T E
FF
EC
TIV
EN
ES
S:
CU
RR
EN
T M
EC
HA
NIS
MS
AN
D D
IRE
CTI
ON
S F
OR
TH
E F
UTU
RE
313. An open environment of information shar-ing and dissemination, fulfilling the publicaccountability and transparency requirementsof a culture of good governance. There are twobroad capacity dimensions relating to this area:
• Establishing participatory results-based mon-itoring and evaluation, particularly at benefici-ary levels; and
• Creating institutions in civil society that haveawareness of results-based monitoring andevaluation in order to inculcate a culture ofscientific public inquiry and accountability.
depend on several factors, including: com-mitment of the senior level leadership; thecapacity of a central agency to promoteresults-based monitoring and evaluation; thecredibility of evaluators/analysts; extensionof the system to places where developmentresults/effects are generated (e.g., line/sectoralagencies, sub-national and grass-roots levels);timeliness; ability to produce results quickly;and robustness of the system (i.e., its ability totransmit data/information reliably from thepoint of origin to the central agency/other rel-evant points.)
• Capacity to convert data, through analyticaland substantive work, for lessons and policysupport. This is clearly critical to the credibil-ity of results-based monitoring and evaluationas a tool for promoting greater developmenteffectiveness. This area also provides a majoropportunity for national and internationaldevelopment cooperation practitioners toform effective partnerships for learning,experimenting, piloting, and work on policyanalysis and development. These practitionersmight include development agencies andresearch and academic institutions of develop-ing and developed countries. Often additionalstudies may also be needed to back up evalua-tion findings.
• Maturity of bureaucracy to accept results-based monitoring and evaluation for system-atic learning and adjust development policiesand processes in line with emerging lessons.
• Political will and commitment – evaluationsare always likely to raise issues of political sen-sitivity for administrations. Capacities shouldbe in place to ensure political ownership andconstructive presentation of sensitive informa-tion in a manner acceptable to stakeholders.
IV.
CO
NC
LUS
ION
S
32
partnership with different actors (e.g., civilsociety, private institutions).
• Establish UNDP as a knowledge-based learn-ing organization where lessons are consolidat-ed and made accessible to all.
• Strengthen national capacity for results basedmonitoring and assessment and learning.
• Intensify the efforts to establish partnershipsamong international and national institutions.
Development is far from being an exact science.Determining the true impact of support provid-ed by an organization is a trial and error processof considerable complexity. In this context, it isimportant that the tools and reforms highlight-ed in this report be seen as informed by thebest, most current knowledge available, whichwill require regular reassessment. Future suc-cess, however, will hinge on effective collabora-tion with partners at the country level, bothwithin government and the donor communityand extending beyond it. While accomplishingthe vision of the Administrator will take sometime, current efforts reflect the commitment ofUNDP to make increased development effec-tiveness a reality.
The commitment of UNDP to developmenteffectiveness lies at the heart of ongoingreforms. The shift to results-based managementand the efforts to strengthen knowledge-shar-ing networks provide the strategic frameworkunder which this commitment has found con-crete application. The overarching goal is tostrengthen the organization’s understanding ofhow successfully it is contributing to positivechanges in people’s lives, and to use this knowl-edge to reinforce the effectiveness of its pro-grammes. At the systemic level, establishingpartnerships with other national and interna-tional institutions to share the lessons learned isincreasingly recognised as a vital component ofan effective development strategy.
However, these reforms remain a work inprogress. There is a need to build upon theprogress, tackling the many challenges high-lighted in this report to enhance the contribu-tion of UNDP to development effectiveness.Next steps might usefully be the following:
• Strengthen the results-based managementframework by embedding it further within theorganization’s culture, at all levels.
• Continue efforts to integrate monitoring andevaluation into the RBM, recognising thatthese processes are mutually reinforcing.
• Bolster feedback mechanisms and ensure thatevaluation findings are:
_ fed back into policy-making and planningdecisions and accounted for in the design ofprogrammes and projects;
_ shared with others, internally, throughSURFs, and externally.
• Promote participatory approaches to the eval-uation process, by broadening the scope of
IV. Conclusions
GLO
SS
AR
Y
33
Finding – factual statement about the pro-gramme or project based on empirical evidencegathered through monitoring and evaluationactivities.
Impact – see "Results."
Input – a means mobilised for the conduct ofprogramme or project activities (i.e., financial,human and physical resources.)
Lesson learned – learning from experience thatis applicable to a generic situation rather than toa specific circumstance.
Monitoring – a continuing function that aimsprimarily to provide programme or projectmanagement and the main stakeholders of anongoing programme or project with early indi-cations of progress, or lack thereof, in theachievement of programme or project objectives.
Objective – purpose or goal representing thedesired result that a programme or project seeksto achieve. A development objective is a long-term goal that a programme or project aims toachieve in synergy with other developmentinterventions. An immediate objective is ashort-term purpose of a programme or project.
Outcome – see "Results."
Outputs – see "Results."
Performance – the extent to which a pro-gramme or project is implemented in an effec-tive, efficient and timely manner.
Programme – a time-bound intervention thatdiffers from a project in that it usually cutsacross sectors, themes and/or geographic areas,
Capacity Development – the process by whichindividuals, organizations, institutions and soci-eties develop their abilities individually and col-lectively to perform functions, solve problemsand set and achieve objectives.
Direct beneficiaries– usually institutionsand/or individuals that are the direct recipientsof technical cooperation aimed at strengtheningtheir capacity to undertake development tasksdirected at specific target groups. In micro-levelinterventions the direct beneficiaries and thetarget groups are the same.
Effectiveness – the extent to which a pro-gramme or project achieves immediate objec-tives or produces its desired outcome.
Efficiency – the optimal transformation ofinputs into outputs.
Evaluation – a time-bound exercise thatattempts to assess systematically and objective-ly the relevance, performance and success ofongoing and completed programmes and proj-ects of an organization, and to package thefindings in an appropriate format.
Feedback – as a process, consists of relevantinformation from monitoring and evaluationactivities, the dissemination of that informationto target users and, most important, the use ofthe information as a basis for decision-makingand the promotion of learning in an organiza-tion. Feedback as a product refers to informationthat is generated through monitoring and eval-uation and transmitted to parties for whom it isrelevant and useful. It may include findings,conclusions, recommendations and lessonslearned from experience.
Glossary
GLO
SS
AR
Y
34Stakeholders – groups that have a role andinterest in the objectives and implementation ofa programme or project; they include targetgroups, direct beneficiaries, those responsiblefor ensuring that the results are produced asplanned, and those who are accountable for theresources that they provide to that programmeor project.
Strategic Evaluation – an evaluation of a par-ticular issue where timing is especially impor-tant owing to the urgency of the issue, whichposes high risks to, and has generated widelyconflicting views from, stakeholders. It aims toadvance a deeper understanding of the issue,reduce the range of uncertainties associated withthe different options for addressing it, and helpto reach an acceptable working agreement amongthe parties concerned.
Success – a favourable programme or projectresult that is assessed in terms of effectiveness,impact, sustainability and contribution tocapacity development.
Sustainability – durability of positive pro-gramme or project results after the terminationof the technical cooperation channelledthrough that programme or project; static sus-tainability – the continuous flow of the samebenefits, set in motion by the completed pro-gramme or project; dynamic sustainability –the use or adaptation of programme or projectresults to a different context or changing envi-ronment by the original target groups and/orother groups.
Target groups – the main stakeholders of aprogramme or project that are expected to gainfrom the results of that programme or project;sectors of the population that a programme orproject aims to reach in order to address theirneeds based on gender considerations, socio-economic characteristics or other factors.
involves more institutions than a project, andmay be supported by different fundingresources.
Project – a time-bound intervention that con-sists of a set of planned, interrelated activitiesaimed at achieving defined objectives.
Recommendation – proposal for action to betaken in a specific circumstance, and includingthe parties responsible for that action.
Relevance – the degree to which the objectivesof a programme or project remain valid and per-tinent as originally planned, or as subsequentlymodified owing to changing circumstanceswithin the immediate context and externalenvironment of that programme or project.
Results – a broad term used to refer to theeffects of a programme or project. The terms"outputs," outcomes" and "impact" describemore precisely the different types of results:
Outputs – tangible products (including serv-ices) of a programme or project that are neces-sary to achieve its objectives. Example: agri-cultural extension services provided to ricefarmers.
Outcomes – results of a programme or proj-ect relative to its immediate objectives that aregenerated by the programme or project out-puts. Examples: increase in rice yield,increased income for the farmers.
Impact – results of a programme or projectthat are assessed with reference to the devel-opment objectives or long-term goals of thatprogramme or project; changes in a situation,whether planned or unplanned, positive ornegative, that a programme or project helps tobring about. Examples: higher standard of liv-ing, increased food security, increased earningsfrom exports, increased savings owing to adecrease in imports.
AN
NE
X I
: TR
EN
D A
NA
LYS
IS-D
ATA
AN
D M
ETH
OD
OLO
GY
35
The portfolio is disaggregated by type of evalu-ation as well as by year of project approval.Terminal and ex-post evaluations are pooled tojudge changes in the performance and successof projects. The classification of evaluations bythe year in which the evaluated projects wereapproved (see Table 1) is used in this analysis,allowing projects of the same generation to bepooled together and better showing changesover time.
In order to assess trends over time the pool hasalso been divided into two groups of years, from1987 to 1991 and from 1992 to 1998. Thisdivision allows the pools of projects to be largeenough for one to see statistically significantchanges; additionally, as 1992 marked thebeginning of a new programming cycle forUNDP, it seemed an obvious point of division.
The portfolio of evaluations in CEDAB hasbeen disaggregated by execution modality toassist in analysing the effects of NEX on theprojects. Since the General Assembly decisionto make NEX the norm in the FifthProgramming Cycle (1992-1996), it hasbecome the executing option of choice for themajority of all projects launched. However,since the pool of evaluation reports in CEDABcontains an older generation of projects, thistrend analysis does not capture the full effect ofUNDP’s commitment to NEX.
UNDP maintains a central database of all of theproject evaluations completed in the organiza-tion. The Central Evaluation Database(CEDAB) contains the evaluators’ work both inthe form of a written summary of their finalreport and their completed project evaluationinformation sheet (PEIS). UNDP had takenmajor steps in building a comprehensive frame-work that ensures substantive accountabilityand allows for systematic monitoring and eval-uation of its performance. Nevertheless, manytools are still in the development stages whileothers need to be updated. For example,CEDAB, PEIS and the Handbook (AHandbook for Programme Managers: Results-ori-ented Monitoring and Evaluation) require fur-ther attention to ensure conformity with theresults-based management framework.
The trend analysis of UNDP project perform-ance looks at various measures of performanceand success to capture the overall effectivenessof the organization. This analysis is organizedaround the criteria established in the Handbookand based on the information provided in thePEIS.
The Portfolio of Evaluations in CEDAB
The full portfolio of evaluations in CEDABcontains over 1,400 reports and accounts for atotal of almost US$3 billion in financial resources.
Annex I: Trend Analysis - Data and Methodology
TABLE 1. DISTRIBUTION BY TYPE OF EVALUATION AND BY YEAR OF PROJECT APPROVAL CONTAINED IN CEDAB (NUMBER OF REPORTS)
1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 Total
Mid-Term 81 65 49 34 31 66 29 14 11 7 15 6 08
Terminal 109 102 79 59 39 55 26 15 10 10 8 2 514
Ex-Post 4 12 7 11 8 6 5 1 3 2 2 0 61
Total 194 179 135 104 78 127 60 30 24 19 25 8 983
AN
NE
X I
I: R
EV
IEW
OF
SU
B-R
EG
ION
AL
RE
SO
UR
CE
FA
CIL
ITIE
S
36
adequate resources. In addition, the review indi-cated that the SURFs encountered difficulty inperforming both technical support services andknowledge management functions at presentstaffing levels. In many instances, despite dis-playing an entrepreneurial spirit that has over-come much adversity, the SURF staff has beenunable to meet demand from the countryoffices, especially in the area of technical back-stopping.
The review’s recommendations address boththe specifics of the SURF system and the over-all institutional context. To meet the growingdemand from country offices for technical sup-port services, the first set of recommendationsincludes a strengthening of the expert capacityat the sub-regional level, which can be accom-plished through the decentralisation of BDPofficials. The review also suggests a number ofways in which the expert referral system and theexisting networking system could be enhanced.Finally, it underlines the need for two addition-al SURFs to cover Latin America and Africaadequately.
At the institutional level, the review teamemphasised the need to embed and empowerthe knowledge management culture withinUNDP. It recommends that the steps requiredto achieve this include the implementation, atthe senior management level, of a coherentvision, the appointment of a Chief KnowledgeManager and the creation of a KnowledgeManagement Council designed to ensure coor-dination and standardisation at all levels tomaximise efficiency. Corollary to these recom-mendations is the need to map UNDP’s knowl-edge needs (what kind of knowledge, where, forwhom), which is especially critical at a time
At the request of the Associate Administratorand in response to the Transition Team report,the Evaluation Office launched a review of theSURF system, UNDP’s primary knowledgemanagement initiative to establish itself as alearning organization. The SURFs began fulloperations in early 1999, as a result of a 1997UNDP decision refocusing the organization’smission and adapting it to future challenges byleveraging technology and new business prac-tices, consolidating its knowledge base andfacilitating the dissemination of information.
The review team found that the SURF systemhad made a significant contribution to countryoffices by providing a series of services that werenot previously available to them. These servicesinclude the delivery of technical support (back-stopping), referrals and access to informationon experts and programmes, the identificationand dissemination of best practices, and thepromotion of networking and informationsharing. In particular, the decentralised struc-ture of the SURFs was viewed as a key elementin their usefulness. In line with the statedknowledge management principle to provide"the best knowledge to the right person at theright time," the review suggests many ways inwhich, in each area of focus, the SURF systemis strengthening the knowledge managementprocess by effectively garnering, consolidatingand sharing relevant information for the coun-try offices.
However, the review highlighted a number ofareas in need of improvement. It was first notedthat the development of the SURFs wasuneven, due in part to the absence of a globalcoherent knowledge management strategicframework at the senior level and to a lack of
Annex II: Review of the Sub-regional Resource Facilities
AN
NE
X I
I: R
EV
IEW
OF
SU
B-R
EG
ION
AL
RE
SO
UR
CE
FA
CIL
ITIE
S
37when UNDP moves towards an upstreamfocus. Finally, among a series of other recom-mendations, the team also suggests that thetechnology itself, and the software in particular,be upgraded to ensure the swiftest possible dis-semination of knowledge.
The report was submitted to the ExecutiveTeam, which will consider how the recommen-dations can be implemented most effectively,and will subsequently be submitted to theExecutive Board. UNDP is currently organiz-ing a workshop with a panel of outside special-ists and in-house stakeholders to further deep-en some of the key findings and move theprocess forward for organizational action.
