assessing multilateral organisation effectiveness: a ... › lotusquickr › cop-mfdr... · common...

Assessing Multilateral Organisation Effectiveness: A Comparative Analysis of Data Collection Tools

Draft

A discussion paper presented to the Multilateral Organisation Performance Assessment Network (MOPAN) Working Group in Stockholm on June 11th, 2007.

Submitted to: Andrew Clark, A/Director

Policy and Strategic Planning Division Multilateral Program Branch, CIDA

Prepared by Werner Meier, Director

http://www.RBMG.ca

Wednesday, June 6, 2007

i

LIST OF ACRONYMS AfDB African Development Bank BO Bilateral Organization COMPAS Common Performance Assessment System CIDA Canadian International Development Agency DAC Development Assistance Committee Danida The Danish International Development Agency DFID UK Department for International Development FAO Food and Agricultural Organization of the United Nations FARP Framework for Assessing Relevance and Performance (MFA - Sweden) IFI International Financial Institution M&E Monitoring and Evaluation MDB Multilateral Development Banks MDG Millennium Development Goals MEFF Multilateral Effectiveness Framework (DFID) MES Multilateral Effectiveness Summary (DFID) MES-BSC Multilateral Effectiveness Summary Balanced Scorecard (DFID) MERA Multilateral Evaluation Relevance and Assessment system (CIDA) MfDR Managing for Development Results MFA Ministry of Foreign Affairs MMS Multilateral Monitoring System (Netherlands) MO Multilateral Organization MOPAN Multilateral Organizations Performance Assessment Network MPA Multilateral Performance Assessment (frameworks) NPM New Public Management ODA Official Development Assistance OECD Organization for Economic Cooperation and Development PCM Project Cycle Management PFM Public Financial Management PMF Performance Management Framework (Danida) PSIA Poverty and Social Impact Analysis RAP RBM Assessment Program (Danida) RBM Results-Based Management SIDA Swedish International Development Agency UNAIDS Joint United Nations Programme on HIV/AIDS UNDP United Nations Development Program UNFPA United Nation Population Fund UNICEF United Nations Children�s Fund WHO World Health Organization

ii

TABLE OF CONTENTS LIST OF ACRONYMS...................................................................................................... i TABLE OF CONTENTS ................................................................................................. ii 1. Introduction.............................................................................................................. 1

1.1 Background..................................................................................................... 1 1.2 Rationale......................................................................................................... 1 1.3 Purpose........................................................................................................... 2 1.4 Limitations....................................................................................................... 2

2.0 Inventory of Assessment Approaches................................................................... 2

2.1 Sorting Apples from Oranges.......................................................................... 2 2.2 Established Corporate Approaches ................................................................ 3 2.3 Recent Survey Initiatives ................................................................................ 3 2.4 New Approach Experimental Models ............................................................. 3

3.0 Comparative Analysis.............................................................................................. 4

3.1 Purpose........................................................................................................... 4 3.2 Design............................................................................................................. 5 3.3 Overall Methodology ....................................................................................... 6 3.4 Key Performance Indicators and/or Questions ............................................... 8 3.5 Transparency and Resource Allocation ........................................................ 10 3.6 General Strengths and Weaknesses ............................................................ 11

4.0 Prospective for a Common Approach.................................................................. 12

4.1 New Public Management Context ................................................................. 12 4.2 Rationale for Assessing Multilateral Effectiveness ....................................... 12 4.3 A Harmonised Approach ............................................................................... 13 4.4 Design Principles for a Common Assessment Approach.............................. 14 4.5 Proposed Common Assessment Approach................................................... 18

ANNEX A: LEXICON OF TERMS USED...................................................................... 22

ANNEX B: SUMMARY OF MOPAN SURVEY METHODOLOGY................................. 23

ANNEX C: SUMMARY OF THE MEFF METHODOLOGY ........................................... 25

ANNEX D: SUMMARY OF THE MES METHODOLOGY ............................................. 28

ANNEX E: SUMMARY OF THE MERA METHODOLOGY........................................... 30

ANNEX F: SUMMARY OF THE SCORECARD METHODOLOGY............................... 32

ANNEX G: SUMMARY OF THE PMF METHODOLOGY ............................................. 34

ANNEX H: SUMMARY OF THE RBM ASSESSMENT METHODOLOGY.................... 38

ANNEX I: SUMMARY OF THE FARP METHODOLOGY ............................................. 40

ANNEX J: CROSS-CASE COMPARISON TABLE....................................................... 42

1

1. Introduction

1.1 Background The Multilateral Organization Performance Assessment Network (MOPAN) is a network of nine1 like-minded donor countries having a common interest in:

(a) sharing information and mutually drawing on their experience in the monitoring and assessment of the work and performance of multilateral organisations;

(b) conducting annual surveys on MOs through their embassies and country offices (the Annual MOPAN Survey); and

(c) carrying out joint evaluations of MOs. The MOPAN member countries jointly conduct an annual in-house survey of multilateral partnership behaviour in developing countries (partnerships with national governments, civil society and other bilateral and multilateral development agencies). The Survey is based on the perceptions of MOPAN member embassies or country offices, arising from their day-to-day contacts with multilateral organizations. The MOPAN Annual Survey2 is not an evaluation and does not cover actual development results achieved.

MOPAN members are using the results of the Annual Survey for their own accountability on multilateral financing and as input: (a) into their policy towards the multilateral organizations concerned; (b) to strengthen their participation in the governance of these organisations; (c) for their joint advocacy work; and (d) to contribute to wider debates on aid effectiveness.

1.2 Rationale At MOPAN meetings in 2006 there was consensus that members required further information on the effectiveness3 of individual multilateral organisations (MOs) beyond the Annual Survey. This was in order to meet growing demands for accountability and in order to make more informed policy choices as multilateral budgets increased.

MOPAN members therefore agreed to establish a Working Group to begin moving towards a common approach for the assessment of multilateral effectiveness which would involve initiatives beyond the Annual Survey. The objectives of the Working Group are two-fold:

! The first is to compare the questions and indicators used in any existing methods to assess agency performance and to develop a common core set that can be used jointly;

! The second is to work with the multilaterals to improve their self reporting based on a clearer understanding of the information required to meet accountability and reporting requirements.

A number of MOPAN members have already developed questionnaires, checklists and scorecards for assessing the organisational effectiveness of their multilateral partners. It was unclear however the extent to which any could serve as the basis for a more harmonised approach in the future, or how multilateral reporting maps on to the information requirements of different donors. The consultant was therefore appointed by the Working Group to provide some independent advice.

1 Current MOPAN members are Austria, Canada, Denmark, Finland, France, the Netherlands, Norway, Sweden,

Switzerland and the United Kingdom. Ireland is currently an observer. 2 A summary description of the survey methodology is provided in Annex B. 3 A lexicon of terms used in this paper, e.g., effectiveness, is provided in Annex B based on the OECD-DAC

�Glossary of Key Terms in Evaluation and Results Based Management�.

2

1.3 Purpose The Consultant was mandated to assist the MOPAN Working Group in fulfilling its first objective by initially conducting a comparative analysis of existing bilateral donor assessment approaches and to recommend how a more harmonised approach might be developed and applied by Working Group members in the future. The specific purpose of this discussion paper is therefore:

! To compare existing bilateral donor approaches to assessing multilateral organisation effectiveness and to provide advice on the feasibility of developing a common approach.

All available documentation provided by the members of the MOPAN Working Group was reviewed and telephone interviews conducted with representatives of MOPAN member organisations. The indicators/questions employed in the assessment tools and survey questionnaires which constituted or formed part of the various assessment approaches were analysed using a concept mapping technique. This grouping and mapping process allowed the consultant to identify the priority areas of assessment of each approach, as well as the common priorities for assessing multilateral effectiveness. Few gaps were identified given the comprehensiveness of some of the assessment tools and the range of assessment priorities.

1.4 Limitations The limitations affecting the quality of the comparative analysis presented in this report are relatively common for this type of work. The allocated levels of effort to conduct the research and analysis, as well as the timelines were fixed parameters within which the work had to be completed. Access to MOPAN members and frequency of consultation opportunities during the research process was also limited, as was the amount of available documentation on some of the assessment approaches reviewed, especially the more recent initiatives. Lastly, the Consultant was appointed by the Working Group to provide some independent advice which will obviously be influenced by past experience and professional preferences.

2.0 Inventory of Assessment Approaches4

2.1 Sorting Apples from Oranges Establishing an inventory of multilateral assessment approaches has been akin to sorting apples from oranges and has led to some surprises as the work has progressed. A number of MOPAN members had been identified as having already developed �questionnaires, checklists and scorecards� for assessing the organisational effectiveness of their multilateral partners. Examples included the Dutch �Multilateral Monitoring Survey�, the Danish �Performance Management Framework� and the British �Multilateral Effectiveness Framework�. While at first blush they may appear to be simple data collection instruments, this is not the case for all, since they range in character from well established corporate approaches to the institutional effectiveness �check list�. Based on implementation experience and lessons learned, in some cases from repeated cycles of data collection, a second generation of assessment approaches have also been developed. Comparisons are rendered even more challenging with the inclusion of relatively new bilateral donor initiatives, such as, the Canadian �Multilateral Effectiveness and Relevance Assessment� and the even more recent Swedish �Framework for Assessing Relevance and Performance of Multilateral Organisations� which have limited supporting documentation and have only recently completed a first cycle of data collection.

4 Summary descriptions of the assessment approaches included are provided in Annexes C-I.

3

Returning to apples and oranges, yes they are classified as fruit, but prudence must be exercised to avoid a false analogy such as where an "apple" is faulted for not being a good "orange." For this reason the assessment approaches being reviewed have been grouped according to some common characteristics described below.

2.2 Established Corporate Approaches The primary criteria for this grouping of established corporate approaches are the number of data collection cycles that have been undertaken and the extent of corporate involvement in the assessment. The Danida Performance Management Framework (PMF) fulfils both criteria since it was first introduced in 2003, has completed a number of annual data collection cycles and systematically involves Ministry of Foreign Affairs personnel in Headquarters (HQ), Regional Delegations and Embassy staff, as well as soliciting the views of other relevant government departments. It should also be noted that the multilateral effectiveness assessment is only one component within the PMF which serves as a management framework for bilateral cooperation programming as well. The Danida Annual Performance Report follows the PMF structure and criteria providing comprehensive performance information. The Netherlands Ministry of Foreign Affairs Scorecard also fulfils both criteria since it was first introduced in 2004 and has completed annual data collection cycles since. It is also a corporate effort which systematically solicits the views of HQ and Embassy staff, as well as relevant policy departments. While the Scorecard has until recently been a confidential document, the Embassy field survey report is posted on the corporate intranet, so that it can be available to everyone at the Ministry.

2.3 Recent Survey Initiatives The primary criteria that characterises this group of recent survey initiatives is that they have only undertaken once cycle of data collection since inception and that they appear to rely on a single survey instrument for data collection. The Swedish Framework for Assessing Relevance and Performance (FARP)5 completed its first round of data collection in 2007 using an Embassy staff survey instrument, while the CIDA Multilateral Effectiveness and Relevance Assessment (MERA) was piloted in late 2006 using a data collection template for use by the responsible Multilateral Desk Officer. In neither case have the reports been fully completed. Included in this group is the DFID Multilateral Effectiveness Framework (MEFF) even though it was first deployed in 2004, it has not been used since. It also relies primarily on a single data collection instrument referred to as the �checklist� by donor and multilateral organisation staff alike. Unlike the other two assessment approaches in this grouping, the MEFF methodology is very well documented and a very thorough analysis of not only the findings, but also the feedback and lesson learned have been documented.

2.4 New Approach Experimental Models What characterises this experimental models grouping is the innovative and unique nature of the assessment approaches, their relative newness and their second generation status. The Danida RBM Assessment Program was launched in 2006 as the �New Approach�. Five (5) highly

5 The title of the Swedish assessment approach was based on an unofficial translation and is not final.

4

focussed and rigorous RBM assessments have been completed of the following UN Agencies: UNDP, UNFPA, IFAD, OHCHR and UNICEF. While an integral part of the overall Danida PMF methodology, the implementation of these assessments remains somewhat outside the Danida corporate structure given that the Dalberg Global Development Advisors has been commissioned to undertake them. The RBM assessment methodology is very well documented and the assessment reports are equally as comprehensive and insightful. First introduced in a June 2006 DFID Fact Sheet, the Multilateral Effectiveness Summaries (MES) was also introduced as a �New Approach�. It employs a Balanced Scorecard (BSC) approach the indicator data fields of which are populated by secondary sources, e.g., MEFF, MOPAN Survey, Paris Donor survey, etc. It is unclear as to whether this is an assessment approach that will be used by DFID going forward, or whether it is a proposal for consideration by other like-minded donors in the spirit of harmonisation. Nevertheless some data collection has apparently taken place recently and Effectiveness Summaries for sixteen (16) MOs will be published in July 2007.

3.0 Comparative Analysis

3.1 Purpose In the 2006 CIDA survey bilateral donors identified accountability and learning, i.e., identifying strengths and weaknesses for corrective action, as the two most important reasons for undertaking multilateral assessments. An analysis of the stated purpose for each of the assessment approaches presented in the table below does not however reveal the same consistency of messaging.

Assessment Approach

Inception Year

Freq. # of MOs / Year Assessed

Purpose

PMF Danida

2003 Annual 3 MOs/yr/country across 15 countries

To: (a) enhance the quality of development cooperation, (b) improve management and continuous learning, (c) strengthening accountability.

Scorecard Netherlands

2004 Annual 25 MOs / year

To assess the relevance of MOs to Dutch policy objectives, their performance and adoption of RBM.

FARP Sweden

2007 Once 28 MOs / yr To inform budget decision-making process, and to provide an analytical basis for strategic work with the MOs.

MERA CIDA

2006 Once 19

To determine effectiveness and relevance with a view to directing future resources to enhance MO performance.

MEFF DFID

2004 Once 23

Provide information for agency�s: (a) Public Service Agreement reporting, (b) its Institutional Strategies, (c) its financing strategies.

RBM Assessment Danida

2006 Upon Request

5

Provide support to: (a) strengthen MOs RBM capacity (b) enabling bilateral donors to depend on MO results reporting for assessments.

MES-BSC DFID

2007 Once 23

Provide information for agency�s: (a) Public Service Agreement reporting, (b) its Institutional Strategies, (c) its financing strategies.

5

The PMF purpose statement is the exception as it focuses on learning and improvement, while the adjunct RBM Assessment, as well as the Scorecard also include an RBM strengthening rationale. These assessment approaches appear to have a dual measurement and management purpose. Accountability is also a prominent message in the purpose statements of the PMF, MEFF and MES but not explicitly stated for any of the other assessment approaches. What is explicit in the latter two cases, as well as for the FARP and MERA is the intention to use the assessment to �inform� or �direct� funding decisions. Relevance to national policy priorities and/or international development goals is also implied in the Scorecard and MERA statements. In summary, the impression given by this cursory review of the above purpose statements is that in the majority of cases the performance measurement function is viewed as a means to demonstrating accountability and influencing funding decisions rather than to encourage organizational learning and change.

3.2 Design Key to understanding the various assessment approaches is their overarching design characteristics, particularly the criteria and/or perspectives which are used to structure data collection, analysis and even reporting. An analysis of the design characteristics for each of the assessment approaches presented in the table below reveals variations around three principal themes: relevance to national and international development priorities, internal performance ( i.e., institutional effectiveness) and external performance (i.e. development effectiveness).

Assessment Approach

Assessment Criteria and/or Perspectives

# of Criteria

National and International Development Priorities Addressed

PMF Danida

(a) Danida Corporate level; (b) MO HQ level; (c) Country level (field level)

3 Intn�l: MDGs, PRSPs and cross-cutting issues, e.g., GE, HRDGG, ENVIR


Scorecard criteria: a) internal performance, b) external performance, and c) relevance

3 Intn�l: MDGs and Paris Declaration; Dutch priorities: e.g., PSD, GGDD, anti-corruption, etc.

FARP Sweden

Relevance: a) goal congruence and b) role relevance. Agency performance c) internal and d) external performance

4 Intn�l: MDGs and Paris Declaration; Swedish priorities: HRGGDD, GE, NR, economic growth, social development and security, etc.

MERA CIDA

(a) relevance, (b) results (c) managing of the institution

3 Intn�l: MDGs and Paris Declaration Canadian priorities: GE, ENVIR, and RBM

MEFF DFID

Eight organizational systems and their focus on (a) internal performance, (b) country level results and (c) partnerships

8 Intn�l: MDGs and Paris Declaration: partnership and harmonisation


RBM System Elements a) Architecture b)Process, and c) Incentives/Resource Allocation

3 N/A

MES-BSC DFID

Four quadrants of the Balanced Scorecard: a) Building for the Future, b) Managing Resources, c) Partnerships d) Country / Global Performance.

4 Intn�l: MDGs and Paris Declaration: partnership and harmonisation

6

The inclusion of relevance as an assessment criterion is either explicit or implicit for the PMF, Scorecard, FARP and MERA. As usual there are two parts to an assessment of relevance, relevance of the MOs development programming to national (donor country) policy priorities and/or relevance to internationally agreed upon development priorities. The specific global, sectoral or thematic priorities are generally identified in the supporting documentation or built into the data collection tools used and have bee presented in the table above.

Internal performance as an assessment criterion is the most common among all the approaches, is used consistently but to varying degrees depending on the importance accorded to it. For example, the MEFF has been critiqued for having been too focused on internal performance issues related to the eight organizational systems. The MERA refers to it as �managing the institution�, the MES-BSC has two quadrants �Building for the Future and �Managing Resources� which are internally focussed, while other approaches make reference to assessing �institutional effectiveness�. The RBM Assessment could be viewed as well as being exclusively dedicated to examining the internal RBM systems and processes, although widely considered precursors for development effectiveness, i.e., external performance.

The internal � external performance dichotomy appears to also be employed synonymously with the MO headquarters versus country-level distinction; external performance being supposedly equated with country-level. Both the PMF, MEFF and MES-BSC make specific reference to �country level� results, however this is not to be mistaken with an assessment of actual development results. In the case of the PMF, Scorecard and FARP, data collection instruments have been specifically designed to collect information from Embassy staff at the country level �about� MO performance, e.g., harmonization, partnership and achievement of development results. The provenance of the data being mistakenly equated with its data properties. In the case of the MES-BSC, as well as the aforementioned, the majority of the �country level� performance questions are at best �proxy� indicators of MO development effectiveness. Consequently, the internal � external performance dichotomy becomes somewhat confusing.

3.3 Overall Methodology The caveat for this comparative analysis of overall methodologies is that it is based on available information. As previously mentioned, some of the assessment approaches are very well documented in terms of their methodologies and some are not given their relative �newness�. Nevertheless, there are some important similarities in methodology among most of the assessment approaches, with the exception of the RBM Assessment (Annex D) and the MES-BSC (Annex H) which are quite unique. For this reason please refer to the appropriate annexes for the latter and the data table on the next page for a comparison of the former.

The PMF and Scorecard are bonafide corporate approaches which use multiple lines of evidence, including an Embassy staff survey to nourish their annual assessments with data. A MFA Desk/Policy Officer, presumably one responsible for each MO serves as the focal point and acts as the �team leader� for the assessment. They fulfill the coordination and quality assurance functions and are supported by other staff positions responsible for compiling the staff questionnaires and Embassy survey data, disaggregating it by MO and providing MO specific summary reports. The findings from documentary content analysis of MO specific annual reports, evaluation reports, or donor evaluation reports all have to be systematically integrated. The Desk/Policy Officer acts as the crucible for synthesizing the data, albeit mostly qualitative, and transforming it into performance information.

7

Assessment Approach

Overall Methodology

Data sources

# of KPI/Qs

Type KPI/Qs

Rating scales used

Scores used to

comparePMF Danida

Assessment lead by MFA Desk Officer using multiple lines of evidence, including Embassy staff survey and RBM assessment

Multiple sources: MFA Desk Officers, Regional Delegations (NY, GEN, WASH, ROME), Embassy staff, MO staff and managers (RBM Assessment)

21 Mixed No No


Scorecard updated by MFA Policy Officer heavily based on opinion surveys and other empirical data.

Scorecard sources: HQ experiences, policy department perceptions MMS Survey, MOPAN Survey, other findings from the field and analysis of MO evaluations.

15 Mostly Evaluative

Yes No

FARP Sweden

Assessment by MFA Desk Officer who leads the team drawing on evaluation reports and the informed opinions of e.g. constituency offices, Delegations and field staff.

Mandatory: Survey of 20 Embassies; HQ input; SIDA thematic input; analysis of audit reports and annual reports. Additional: MOPAN reports; MO evaluations, peer reviews, other donors.

33 Evaluative Yes No

MERA CIDA

Assessment conducted by CIDA MPB Directorate staff drawing on personal perceptions and available empirical data.

CIDA staff, available documentation and MO performance reports.

26 Evaluative Yes Yes

MEFF DFID

Assessment initially conducted by DFID staff with various degrees of input from MOs and ending with final dialogue meeting.

DFID staff with various degrees of input from MOs staff and managers

72 Empirical Yes Yes


Assessment of MOs RBM systems in HQ/field conducted by consultants in a highly collaborative manner.

Multiple sources: MO documentation, staff interviews, on-site observations, demos of online data systems, discussion workshops interviews with staff.

33 Mixed No No

MES-BSC DFID

Assessment initially conducted by DFID staff using various secondary data sources.

Secondary data sources: MEFF, MOPAN Survey, Paris Survey, DAC peer reviews, MO performance and evaluation reports.

40 Mixed Yes No

8

The MERA and the FARP have similar, or potentially similar corporate approaches to the PMF and Scorecard, but are perhaps still too recent to have well established data gathering and analysis procedures and processes in place which generally come after repeated annual cycles. A well structured process of mandatory and additional data sources has been outlined for the FARP which will no doubt add to the rigor of this assessment in the long term. On the other hand, it appears that the MEFF and to a certain extent the MERA depend inordinately on the opinion of the responsible Desk Officer, initially or otherwise, to complete the summary �checklist� or �template�. The extent of counter-verification of the assessment with the MO also varies from none � to extensive, depending on the circumstances and preparedness of the MO to get involved. Both the FARP and the MERA have the MO assessments reviewed internally by management bodies as a form of face validity check and presumably to smooth out any extreme opinions or question any outlying data points.

3.4 Key Performance Indicators and/or Questions There are collectively almost 250 key performance indicators and/or questions (KPI/Qs) being used to assess multilateral effectiveness based on the above data table. Most or all of the KPI/Qs for the Scorecard, FARP and MERA are evaluative in their formulation, i.e., they require the respondent to express an informed opinion or to exercise expert judgment based on available information. While this methodology has merit, there are issues of trustworthiness when respondents change over time, e.g., new inexperienced staff, or personality conflicts arise.6 It should also be noted that the addition of rating scales to evaluative questions does not enhance their trustworthiness, unless of course the scales are criterion-referenced. The MEFF provides a good example of the use of empirical KPI/Qs in conjunction with a rating scale that requires supporting evidence in the form of text segments. Unfortunately, the specificity of such an assessment approach requires the use of many more KPI/Qs in order to cover the same breadth of evaluation topics or assessment priorities.

Ideally, a mixed set of evaluative and empirical KPI/Qs would generate the qualitative and quantitative data that if properly cross-referenced or triangulated, would produce credible performance information. Based on the limited information available, such appears to be the case for the Danida PMF combined when combined with the RBM Assessment, as well as the DFID MES-BSC which incorporates MEFF data. In both cases there is a balance between evaluative data obtained from various opinion surveys and empirical data based on observations, document content analysis and other factual information.

Determining the extent of overlap and gaps in the selection of the KPI/Qs is part of this comparative analysis and was accomplished using a concept mapping technique. First, the KPI/Qs were extracted from the primary data collection tools used in each assessment approach. They were then pooled and sorted based on easily recognized common characteristics. This technique resulted in the identification of four priority areas and a number of sub-priority clusters for MO assessment which serve as a frame of reference for determining the extent of overlap and gaps. The structure of the frame of reference is open to interpretation and the labels used are a matter of personal preference. The final product of the concept is the distribution table presented on the next page where the KPI/Qs are mapped against this frame of reference.

6 The terms merit and trustworthiness are used as synonyms for validity and reliability respectively when

discussing the characteristics of qualitative data. �Fourth Generation Evaluation� by Yvonna Lincoln and Econ Guba, Econ (1989).

9

From the perspective of individual assessment approaches, the gaps are self-evident, while from a more collective perspective there are only two obvious cases. The Danida RBM Assessment (RAP) suggests the importance of examining MO use of corporate indicators and targets. The two DFID assessment approaches (MEFF and MES-BSC) suggest the importance of examining quality control and quality assurance processes put in place by MOs prior to the approval of new development investments, as well as management systems for assessing and tracking the quality of country portfolio investments. The implicit suggestion, with which I would concur, is that the related KPI/Qs are good proxy or predictive indicators of MO effectiveness.

The extent of overlap is equally self-evident. It should be noted that the three donor �harmonization� categories could be collapsed, or alternately, more differentiation introduced in the formulation of the KPI/Qs grouped under �harmonization � general�. In any case, the next step in a concept mapping process is prioritization where the stakeholders would rank the importance of each indicator within the sub-priority clusters. This step would serve to identify those KPI/Qs collectively most valued by the stakeholders which would then be used to populate a common assessment tool.

Assessment Approach / Priority Assessment Areas

PMF MMS Scorecard

FARP MERA MEFF RAP MES-BSC

STRATEGIC MANAGEMENT Corporate Governance 1 2 10 4 9 - 2 Corporate Strategy 6 4 5 6 7 2 - Country Strategies 1 4 1 2 - - - Corporate Indicators/Targets - - - - - 7 - Total Number of KPI/Qs 8 10 16 12 16 9 2 Percentage of Total KPI/Qs 38% 67% 48% 46% 22% 27% 5%

OPERATIONAL MANAGEMENT Financial Resources 6 1 2 3 8 4 6 Human Resources 3 - 3 - 9 3 2 Quality @ Entry - - - - 4 - 1 Country Portfolio - - - - 5 - 3 Total Number of KPI/Qs 9 1 5 3 26 7 12 Percentage of Total KPI/Qs 43% 7% 15% 12% 36% 21% 31%

RELATIONSHIP MANAGEMENT Harmonisation w/Multilaterals - 1 - 1 - - - Harmonisation w/ Bilaterals - 1 1 1 - - - Harmonisation � General 1 - 2 1 6 - 4 Alignment w/ Partners 1 - 2 - 2 - 13 Total Number of KPI/Qs 2 2 5 3 8 0 17 Percentage of Total KPI/Qs 10% 13% 15% 12% 11% 0% 44%

KNOWLEDGE MANAGEMENT RBM Stewardship 1 2 1 2 2 - 1 Performance Monitoring - - 1 - 7 4 2 Evaluation Management 1 - - 4 1 1 4 Performance Reporting - - 1 2 10 8 - Lessons Learned - - 4 - 2 4 1 Total Number of KPI/Qs 2 2 7 8 22 17 8 Percentage of Total KPI/Qs 10% 13% 21% 31% 31% 52% 21% Total Number of KPI/Qs 21 15 33 26 72 33 39

10

3.5 Transparency and Resource Allocation Based on the information presented in the data table below, it appears that several bilateral donors intend to use the findings of their MO assessment exercises to inform decision making with regard to resource allocations. At the same time a good number of the same donors do not share, or intend to share, their assessment findings with the MOs in the countries, or otherwise. Furthermore, there are several that do not make even the summary report publicly available.

The relationship between transparency and resource allocation is a critical one for donors to manage if they want to avoid excessive �gaming� and enhance the level of confidence and trustworthiness accorded to the assessment findings. The Monterrey Conference cited transparency as a key principle underlying reforms in the international financial architecture to enhance stable financing for development and poverty eradication. �It is important to promote measures in source and destination countries to improve transparency and the information about financial flows.�7 The assessment approaches which are conducted and reported in an open and transparent manner, e.g., the MEFF, RBM Assessment and MES-BSC, can more easily justify the use of the findings for resource allocation purposes.

Assessment Approach

Report shared with MOs in country

Summary Report publicly available

Report used for Resource Allocation

PMF Danida

No Yes Not mechanically or systematically, but can influence senior management decision-making.


No No - Summary report on each MO based on MMS data is given to Policy Officer which is then inputted into the Scorecard which has not been a public document.

Not mechanically or systematically, but can influence policy dialogue and funding decisions.

FARP Sweden

No, but the intention is to provide feedback, in one form or another, to MO during high level consultations.

Not yet decided for the future � for the first year it will not be.

Intended to guide budget allocations and funding decisions, but not mechanically so. Will influence policy dialogue.

MERA CIDA

No No Yes, to reward good performance with additional funds.

MEFF DFID

Yes Yes Yes


Yes Yes No

MES-BSC DFID

N/A Effectiveness Summaries for 16 MOs will be published in July 2007.

No, rather to inform decision making and to ensure senior managers have a consistent picture of each multilateral.

7 Monterrey Consensus on Financing for Development, (March 2002, pp.10)

11

3.6 General Strengths and Weaknesses The strength of any assessment approach, or evaluation for that matter, lies in the extent to which it is designed to fulfill both its measurement and management functions. Measurement requires multiple lines of evidence built into the design so as to solicit an appropriate balance of evaluative and empirical data from multiple data sources. Opportunities to compare similar data sets received from different data sources, conduct face validity checks, or discuss self-assessment findings are evaluation techniques that enhance the reliability and trustworthiness of the data, findings and ultimately the conclusions. Management requires that attention be paid to MO stakeholder engagement in the assessment process to enhance the credibility and utility of the findings for the multilateral organisation. An appropriate balance must be struck between measurement and management if the dual objectives of accountability and organisational learning are to be respectively achieved.

While the established corporate approaches appear to have some of these characteristics, their weaknesses lay in an over-reliance on opinion-based data and a lack of transparency. With the exception of the RBM Assessment, the remainder appear to favour a measurement approach but are for the most part still in the early stages of development and could benefit from a more balanced and rigorous data collection and analysis methodology, or at least one that is more explicitly documented. Despite their level of complexity, the MES-BSC and RBM Assessment have the greatest potential for evolving into a common approach for assessing multilateral effectiveness.

Assessment Approach

Measurement or Management

Complexity of Use Sustainability

PMF Danida

Mixed measurement and management

Complex given three (3) levels, variety of data sources and collection methods. Embassy survey is fairly simple but time consuming to analyse for individual MOs.

Yes



Scorecard somewhat complex because of variety of data sources. MSS easy for Embassies to answer but time consuming to analyse for individual MOs.

Yes

FARP Sweden

Measurement Fairly simple survey approach, but somewhat complex given it was the first time. Also, rather time-consuming,

Yes

MERA CIDA

Measurement Simple for Desk officers to answer questions with no additional primary data collection requirements.

Yes

MEFF DFID

Measurement Fairly simple to complete Checklist, even for MOs, however data analysis / reporting become complex and time consuming

Not for a single donor



Complex and time consuming to conduct requiring between 60-100 p/d level of effort for two people over three (3) month period.


MES-BSC DFID

Measurement Somewhat complex to compile secondary data given the variety of data sources and timing, perhaps less time consuming to do data analysis for individual MOs.


12

4.0 Prospective for a Common Approach

4.1 New Public Management Context The origins of the New Public Management (NPM) in the benchmark cases of the United Kingdom, Australia, New Zealand, USA and eventually Canada can be traced to shifts in economic policy toward fiscal austerity due to burgeoning national deficits and unsustainable government operating budgets. The over-bureaucratisation of government and organizational inefficiency in the public sector took center-stage on government policy agendas during the 1980s-90s, and for quite sometime thereafter in many other countries. This led directly to changes in public management policymaking, institutional rules and government systems in the areas of expenditure planning, financial management, embracing the concepts of economy, efficiency and effectiveness. The agenda-setting process in public policy including the adoption of performance-based budgeting was greatly affected by this �conjuncture-économique�, seeing major cutbacks in government spending, especially in the social support and healthcare delivery sectors. The long term effects, at least based on the Canadian experience have not been commendable and the growing consensus in the public administration literature is that the NPM has not lived up to the hype. While the genesis of the NPM in Western Europe may have been somewhat different than the Anglo-American benchmark cases, one of its defining characteristics of a more results-oriented approach to public management has remained intact. As the budget cutting and slashing subsided, there has been a fundamental shift in emphasis from efficiency, e.g., �doing more with less�, to effectiveness, i.e., actually �achieving expected results�. This orientation toward the achievement of results, rather than merely managing funds and following correct procedures has been a positive outcome of the NPM era; the �managing for results�, rather than �managing by results� language reflects this post-NPM thinking. However, managing for results requires a fundamental change in organizational management culture which, based on recent experience, is best nurtured in a climate of openness, transparency and partnership.

4.2 Rationale for Assessing Multilateral Effectiveness Like any other RBM initiative, the key incentives for bilateral donors to assess the effectiveness of multilateral organisations are: 1) accountability and 2) organisational learning, not only for the bilateral donors, but also for the multilateral organisations involved. Resource allocations or performance-based budgeting (PBB) should be of lesser immediate importance, especially given the nascent development of solid RBM systems and processes within most MOs, as well as the enormity of the political considerations when determining financial contributions.

Two important dynamics have to be kept in mind. First, just because an MO has not developed a results-based performance reporting capacity doesn�t mean that it is not achieving development results. To reduce funding in such circumstances would be an injustice to the intended beneficiaries and clearly counter-development. Second, just because an organisation is/is not achieving development results, doesn�t mean that the political considerations shouldn�t take precedence. This is the nature of democratic societies where performance information is only one of several considerations in what is inevitably a political decision. For at least these two reasons, it would be inappropriate to elevate resource allocation any higher that a distant third position after organisational learning as a purpose for assessing multilateral effectiveness; at least until all other options have been exhausted.

13

Notwithstanding the above, assessing multilateral effectiveness will really only be cost-effective in the long term if there are going to be consequences for either: 1) not establishing the necessary RBM culture to enable a determination development effectiveness, or 2) clearly demonstrated ineffectiveness. What other consequences would be appropriate in the short-term, and how long does one wait for signs of organisational learning before using intimidation and eventually budget cuts? These decisions clearly require management judgement on a case-by-case basis and hopefully in concert with other like-minded donors - thus the need for a harmonised approach.

4.3 A Harmonised Approach While the Paris Declaration focuses primarily on donor harmonization of common arrangements at the country level to reduce donor and partner transaction costs and improve aid effectiveness, the spirit, intent and commitments apply equally as well to multilateral development programming. Of particular importance is the donor commitment to �work together to harmonise separate procedures�. However, the recent proliferation of bilateral donor initiatives to assess multilateral effectiveness has increased donor transaction costs with every additional survey questionnaire forwarded to Embassy staff and the additional resources required to analyse and report on the findings. Similarly, MO transaction costs will increase as well with every new �checklist�, survey, information requirement and high level management debriefing requested by donors in order to complete their assessments.

One of the lessons learned from the MEFF experience8 was the high transaction costs that these multilateral assessments incurred for both donors and MOs alike. The feedback provided by MOs (Box 8) speaks to the need for a common assessment approach. Progressive bilateral donors with several years of experience are reconsidering their assessment approaches. �As the MOPAN survey has developed into a useful and seemingly sustainable tool, the added value of the annual stand-alone assessment by Danish Embassies should be considered. A harmonized approach to assessing both country-level and corporate performance of multilateral organisations seems to be the way forward.�9 There are sufficient similarities and commonalities in the assessment approaches reviewed in this report, that the questions being asked should no longer be �why� or �if�, but moreover �how�?

8 �The MEFF Methodology: A review of DFID�s multilateral effectiveness framework� by Allison Scott

DFID, IDAD, March 2006. 9 Danida Annual Performance Report 2007, pp 62.

14

4.4 Design Principles for a Common Assessment Approach An appropriate starting point for a discussion on the design principles for a common assessment approach would be the five Managing for Development Results (MfDR) principles agreed upon during the Second Roundtable on Managing for Results in 2004. 10

4.4.1 Focusing the Dialogue on Development Results If the mandates, mission statements, corporate and country strategies of multilateral organisations all espouse the achievement of development results, then why are there still key deficiencies in MO performance reporting that hinder effective performance assessment? A recent survey of bilateral donors 11 identified the following deficiencies in order of importance: 1) poor reporting on development results, 2) poor reporting on country results, 3) unclear application of lessons learned, 4) poor reporting on capacity development results, 5) absence of sufficient on-line data, and 6) limited monitoring and evaluation reporting. In a context of managing for results, credible performance reporting is a by-product of good management, and conversely, its absence is indicative of the organisation�s results-based management culture. Perhaps the answer to the above question lay in the continued adherence to conventional notions of accountability.

Conventional definitions of accountability, whether it be mandate fulfillment or contract execution emphasize compliance with agreed rules and standards in conducting the work of the organization, the prudent use of public funds, fair and honest reporting, etc. While not unimportant, results-oriented management would focus less on input and process management and more on achieving and demonstrating the achievement of results. In 1998, the Government of Canada adopted the following definition of accountability which may be of use:

Accountability is a relationship based on the obligation to demonstrate and take responsibility for performance in light of agreed expectations. 12

This definition works well in non-hierarchical relationships and decentralized circumstances where there is a need to balance greater decision-making flexibility and autonomy with enhanced accountability for results. Most importantly to this discussion is the �obligation to demonstrate� performance, i.e., in reporting on the achievement of development results. The associated indicators of effective accountability are: clarity of roles and responsibilities, clarity of performance expectations, balance of expectations and capacities, credibility of reporting, and the reasonableness of review and adjustments.

Why this seemingly esoteric discussion on accountability? Well simply because the dialogue on development results has to occur within a context of corporate governance to be effective. Since the bilateral donors are shareholder representatives on the governance bodies of multilateral organization, as such, they have an oversight responsibility to hold the organisation accountable.

Design principle:

1. The assessment approach should generate relevant and credible information to support bilateral donors in fulfilling their governance responsibilities for holding multilateral organizations accountable for managing for results.

10 �MfDR Principles in Action: Sourcebook on Emerging Good Practices� (OECD 2005). 11 �Bilateral Methodologies for Assessing Multilateral Performance: Survey Findings� (CIDA 2006). 12 �Modernizing Accountability Practices in the Public Sector�, a joint paper by the Office of the Auditor

General of Canada and the Treasury Board Secretariat of Canada (1998). http://www.tbs-sct.gc.ca/rma/account/oagtbs02_e.asp#Current

16

4.4.2 Aligning Monitoring and Evaluation with Results One of the common observations from implementing results-based management is captured in the saying �what gets measured gets managed� and therein lay the risks associated with designing an assessment tool. Bilateral donors, among many others, would like to see MOs align their programming, monitoring and evaluation systems with relevant development results. As the quotation in the adjacent text box suggests, to reduce the transaction costs associated with conducting assessments of multilateral effectiveness it would be preferable to be able to rely on the performance information provided in the annual reports. Apply the above aphorism proactively and this implies an emphasis on assessing the status of RBM system development and the quality of annual performance reports. As this report has already pointed out, Danida�s RBM Assessment approach has made considerable strides in developing a methodology which serves the dual purpose of generating credible assessment information, while focussing MO management�s attention on areas of critical long term importance to bilateral donor assessment objectives.

Design principle:

2. The assessment approach should generate useful information to support multilateral organizations in their efforts to foster a results-based management culture and develop the requisite monitoring and evaluation systems to facilitate organisational learning, improve development effectiveness and produce credible performance reports.

4.4.3 Keeping Measurement and Reporting Simple A key lesson learned in implementing results-based management has been, �keep it simple�, especially in the initial design stages of a new performance monitoring assessment system. Such advice is not new and has been expressed by the OECD and the World Bank in several publications as reported by Andreas Obser, "[O]ne way of doing so is to replace enthusiasm by practicality and to reduce complexity by identifying <minimum conditions> for credible and quality multilateral organisation performance assessments <under real-world constraints>�.13 The 2004 MEFF experience provides some guidance on this topic wherein the staff time, DFID and MO alike, to complete the �checklists� was considered reasonable, it was the data analysis and interpretation which took a considerable amount of time and resources. While on-line survey technology may offer a partial solution, it is still important to focus on the �need to know� as opposed to the �nice to know� assessment information. There will always be opportunities to improve, refine and add indicators after subsequent rounds of data collection.

Design principle:

3. The assessment approach should apply the �keep it simple� principle, keeping the initial number of indicators reasonable and designing the data collection tools with a view to facilitating data analysis and interpretation. 13 Multilateral Organisations Performance Assessment: Opportunities and Limitations for Harmonisation

among Development Agencies, (GDI 2007)

The focus on and assessment of results-based management in multilateral organisations needs to continue in the coming years to allow Denmark and other donors to fully rely on the organisation�s own results-reporting systems.It could be useful to concentrate on those organizations that are relative laggards in the area of RBM, and to do so through a stronger, more inclusive multi-donor effort. It is necessary for donors to join forces in order to ensure harmonisation of reporting requirement as well as a common understanding of results systems. (Danida Annual Performance Report 2007, pp 61)]

17

4.4.4 Managing For, not by, Results

While many issues could be raised for discussion under this principle, suffice it to say that most of the assessment approaches reviewed in this report were exercises in pure measurement. For example, the MEFF was designed as a measurement �checklist� to be initially completed by the responsible DFID Desk Officer. However, the flexibility shown in its administration garnered some interesting feedback when the �checklist� was jointly completed or used as a self-assessment tool. Apparently, this application of the �checklist� generated a more accurate assessment in some cases, and more importantly incited the MOs to engage in a corporate process of self-reflection. This is a significant accomplishment, even if it only occurs once a year.

Design principle:

4. The assessment approach should be designed as a collaborative management exercise, as opposed to a bilateral donor measurement exercise.

4.4.5 Using Results Information for Learning and Decision Making Bilateral donor agencies can collectively create greater demand for more credible and balanced performance information with a common assessment approach. Not only is there strength in numbers, but speaking with a common voice about the same results information can serve as a powerful incentive for MOs to take the assessment findings seriously. Unlike some of the assessment approaches reviewed in this report, opportunities should be included to collectively review the assessment findings in an atmosphere of openness and transparency, to discuss differences of perspective or opinion without recrimination and to engage in collaborative learning. Depending on the circumstances, final dialogue meetings or high level annual bilateral consultations with the MOs, could serve to develop or review plans of action to redress weaknesses and establish milestones. Opportunities for active management decision making by the multilateral organisations should be included as part of the assessment process in order to foster an RBM culture and develop the requisite systems to meet bilateral donor demand for credible performance information. The Danida RBM Assessments appear to have successfully included this action planning component into the assessment design by not only presenting recommendations but also working with management to identify reasonable timelines and resource requirements for their implementation.14

Design principle:

5. The assessment approach should be designed to engage multilateral organisations in a learning and improvement process.

14 �Assessing Results Management at UNICEF� (Dalberg 2007).

�The process showed a high degree of collaboration, trust and mutuality. UNIFEM would want to convey great appreciation for the process, which is consistent with DFID�s overall approach of valuing learning and self-assessment as a part of enhancing effectiveness�- UNIFEM on the MEFF experience

�We welcome this assessment very much as in the process it has helped us to be more self-critical, as well as given us the confidence to continue to build on the institutional reforms that will make the ADB truly more effective as a development partner in the delivery of aid to our RMCs.� - AfDB on the MEFF experience

18

4.5 Proposed Common Assessment Approach 4.5.1 Shareholder versus Stakeholder Bilateral donors have been referred in this report as shareholder representatives, that is, representatives of the taxpayers in their respective constituencies who contribute the funds to support the multilateral organizations. As shareholder representatives they sit on the governance bodies of these organizations and presumably hold them accountable not only for the prudent stewardship of the funds that we entrusted to them, but also for their performance, as well as the demonstration thereof. The distinction between the role of shareholder and that of a stakeholder presented below was made quite cogently in an �Independent Evaluation of SDC's Interaction with the UNDP� commissioned by the Swiss Agency for Development and Cooperation (SDC) (August 2003).

Shareholder: As a member country of the UNDP and through its financial contribution, Switzerland is a shareholder that contributes to and shares responsibility with other member countries for UNDP as a whole. Under the shareholder perspective, the organisation as a whole is in the foreground.

Stakeholder: As a stakeholder, Switzerland has development objectives and other national objectives. Under the stakeholder perspective, it is this Swiss agenda that is in the foreground. 15

The common assessment approach being proposed hereinafter will require bilateral organizations to forego their stakeholder role in favour of a shareholder role. The reasoning is quite simple; a common assessment approach must put the multilateral organisation in the foreground, while the uniquely national development objectives of each bilateral donor country are put in abeyance. Otherwise, the prospect of coming to agreement on a common assessment tool that also meets the �keep it simple� principle would be quite challenging.

4.5.2 A Common Set of Indicators The comparative analysis of the assessment tools under review included a combined total of almost 250 KPI/Qs, with the MEFF checklist representing the largest sub-group of 72 empirically formulated questions. The concept mapping technique revealed a significant degree of commonality in four priority areas: strategic management, operational management, relationship management and knowledge management. Coincidently, these areas also map relatively well against the new approach DFID MES-BSC, i.e., managing resources, partnerships, building for the future, and country/global performance. Provided with a facilitated priority setting exercise, and given a pool of over 250 KPI/Qs to choose from, the MOPAN members should be able to agree on a core set of common indicators to populate a common assessment tool. More important than the actual indicators selected, is the process that is used to select them. A participatory decision making process that builds consensus, commitment and ownership of the indicators selected would be recommended for all the same reasons that such processes are used in country development programming.

15 Authors: Jens-Eric Torp, TB Consult, Copenhagen and Fritz Sager, Büro Vatter, Berne.

19

4.5.3 Multiple Lines of Evidence The most common refrain among MOs regarding bilateral donor initiatives to assess multilateral effectiveness, aside from them not being harmonized, is that they are superficial, subjective and insufficiently sensitive to specific contexts, i.e., organisational and country. While there is some merit in these comments, especially with respect to primarily single source and single instrument methodologies, there is no reason to expect the same if multiple lines of evidence are built into the methodology of a common assessment approach. The present proposal would build on the strengths of the existing assessment approaches and use modified tools to collect data from at least three data sources representing different perspectives on the same indicators of interest. It would avoid seeking the same indicator data from all data sources for reasons of data validity and in the interest of keeping the approach simple. The following data sources would be used depending on the priority areas for assessment as per the Preliminary Performance Assessment Plan presented below:

! Multilateral Organisation Headquarter Staff (MO HQ Staff) ! Multilateral Organisation Country Office Field Staff (MO Field Staff) ! Multilateral Organisation Site Visit (MO HQ & Field) ! Donor Staff in Headquarter Offices and Delegations (Donor HQ-Del. Staff) ! Donor Country Office Field Staff (Donor Field Staff) ! Country Partner Staff

Preliminary Performance Assessment Plan

Priority Areas Data Sources Data Collection Technique Frequency

1) MO HQ Staff Simplified Self-Assessment Checklist Annual

2) Donor HQ-Del. Staff Simplified Checklist Annual

Strategic Management

3) MO HQ & Field RBM Assessment As Required

1) MO HQ Staff Simplified Self-Assessment Checklist Annual

2) Donor HQ-Del. Staff Simplified Checklist Annual Operational Management


1) MO Field Staff Modified MOPAN Self-Assessment Checklist Cyclical

2) Donor Field Staff Modified MOPAN Survey Paris Donor Survey

Cyclical Annual

Relationship Management

3) Country Partner Staff Modified MOPAN Survey Cyclical

1) MO Field Staff Modified MOPAN Self-Assessment Checklist Cyclical

2) Donor Field Staff Modified MOPAN Survey Cyclical Knowledge Management


4.5.3 Assessment Process Strategic and operational management lend themselves well to a rapid empirical assessment that can be done by donor and MO HQ-Del. staff using a criterion-referenced �checklist� approach once a year. An annual meeting to compare notes and discuss differences would serve as a validity check and be recommended.

Relationship management lends itself well to qualitative assessment. The MOPAN survey process

20

for donors would be continued, but expanded to cover some areas of knowledge management related to managing for results at the country level. The Paris Donor Survey could be used as a reliability check on overlapping indicators. In addition, comparative perspectives on the same indicator set from the MO Field Staff and Country Partner Staff would be advised. Again, a meeting to compare notes and discuss differences between the MOPAN donor representatives in-country and the MO Field Staff would serve as a validity check and be recommended. Since it is clearly not feasible to asses every MO every year in every country, the assessment process would follow the current cyclical pattern of selecting a group of MOs and the countries in which the assessments would focus for that year.

Knowledge management lends itself best to a mixed empirical and evaluative assessment requiring a much broader range of data gathering techniques as employed in the conduct of the Danida RBM Assessments described above. This intensive examination of an MOs RBM culture and systems would be employed on an �as required� basis depending on known factors regarding the MOs included in the annual assessment. For example, MOs known to have been slow to adopt the RBM approach should undergo an RBM assessment at the earliest opportunity. It would not however have to be repeated for a few years thereafter to allow for the organizational change process to take place. Those MOs already capable of some evidence-based reporting on development outcomes could undergo a lighter version of the RBM assessment process. Once an MOs performance reporting meets the agreed upon quality standards there would be no further need for an RBM assessment.

These are the proposed initial building blocks of a common assessment process which understandably could change in light of new concerns or as conditions and priorities change.

4.5.4 Consider Using the Balanced Scorecard The Balanced Scorecard is a strategic management approach developed in the early 1990's by Drs. Robert Kaplan and David Norton from the Harvard Business School. It builds on some key concepts of previous private sector management ideas such as total quality management, including customer-defined quality, employee empowerment, continuous improvement, organisational learning. When adapted appropriately to the public sector, it can be used effectively to visualise assessment priorities, related performance information and to track transition management and performance.16

The DFID application of the Balanced Scorecard for its Quarterly Management Board Reports is a foreshadowing of how this approach could be used in the context of assessing multilateral effectiveness. Again, quite by coincidence, this consultant had arrived at the same conclusion prior to seeing the DFID MES-BSC and had prepared a model for consideration by the MOPAN members (see below). While both models are worthy of serious consideration, what is of greater importance is the process that the MOPAN members will undertake to either modify, adapt and customise either one, or engage in a creative process and start anew to ensure the requisite buy-in and commitment of its members to a common assessment tool.

16 �Balanced Scorecard: Step-by-step for government and nonprofit agencies�, Paul R. Niven (2003).

21

June 3, 2007 16

STRATEGICSTRATEGICMANAGEMENT

GovernanceCorporate Strategy

Country-Level StrategiesCorporate Indicators/Targets

KNOWLEDGEKNOWLEDGEMANAGEMENT

Organisational StewardshipPerformance MonitoringEvaluation ManagementPerformance Reporting

Lessons Learned RELATIONSHIPRELATIONSHIPMANAGEMENT

Donor Harmonisation w/ Multilaterals and Bilaterals

Alignment withCountry Partners

OPERATIONALOPERATIONALMANAGEMENT

Financial ResourcesHuman Resources

Quality @ EntryCountry Portfolio

MfDR

The MOPAN Balanced ScorecardThe MOPAN Balanced Scorecard

22

ANNEX A: LEXICON OF TERMS USED The key terms used in this paper17 are defined as follows: Accountability: Obligation to demonstrate that work has been conducted in compliance with agreed rules and standards or to report fairly and accurately on performance results vis-a-vis mandated roles and/or plans. This may require a careful, even legally defensible, demonstration that the work is consistent with the contract terms. Note: Accountability in development may refer to the obligations of partners to act according to clearly defined responsibilities, roles and performance expectations, often with respect to the prudent use of resources. For evaluators, it connotes the responsibility to provide accurate, fair and credible monitoring reports and performance assessments. For public sector managers and policy-makers, accountability is to taxpayers/citizens.

Development Intervention: An instrument for partner (donor and non-donor) support aimed to promote development. Note: Examples are policy advice, projects and programs.

Development objective: Intended impact contributing to physical, financial, institutional, social, environmental, or other benefits to a society, community, or group of people via one or more development interventions.

Effect: Intended or unintended change due directly or indirectly to an intervention.

Effectiveness: The extent to which the development intervention�s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Note: Also used as an aggregate measure of (or judgment about) the merit or worth of an activity, i.e. the extent to which an intervention has attained, or is expected to attain, its major relevant objectives efficiently in a sustainable fashion and with a positive institutional development impact. Related term: efficacy.

Impacts: Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended.

Institutional Development Impact: The extent to which an intervention improves or weakens the ability of a country or region to make more efficient, equitable, and sustainable use of its human, financial, and natural resources, for example through: (a) better definition, stability, transparency, enforceability and predictability of institutional arrangements and/or (b) better alignment of the mission and capacity of an organization with its mandate, which derives from these institutional arrangements. Such impacts can include intended and unintended effects of an action.

Lessons learned: Generalizations based on evaluation experiences with projects, programs, or policies that abstract from the specific circumstances to broader situations. Frequently, lessons highlight strengths or weaknesses in preparation, design, and implementation that affect performance, outcome, and impact.

Outcome: The likely or achieved short-term and medium-term effects of an intervention�s outputs.

Performance: The degree to which a development intervention or a development partner operates according to specific criteria/standards/ guidelines or achieves results in accordance with stated goals or plans.

Results: The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention. Related terms: outcome, effect, impacts.

17 From the �Glossary of Key Terms in Evaluation and Results Based Management� published by the

Organisation of Economic Cooperation and Development (OECD), Development Assistance Committee (DAC) Working Party on Aid Evaluation. OECD, Evaluation and Aid Effectiveness #6, May 28, 2002.

23

ANNEX B: SUMMARY OF MOPAN SURVEY METHODOLOGY18 Background Since 2003, the Annual Survey covers 3-4 multilateral organisations and is conducted in 8-10 countries. The first Annual Survey was implemented in 2003 as a pilot followed by three full fledged Annual Surveys in 2004, 2005 and 200619. The organizations selected for the 2007 survey are the UNDP, WHO/PAHO and AfDB. The survey will be conducted in Benin, Egypt, Ethiopia, Mali, Senegal, Zambia, Bangladesh, Serbia, Bolivia, Brazil and Nicaragua.

Purpose The objectives of the MOPAN Annual Survey may be summarized as follows:

! better information and understanding of multilateral organizations, their roles and performance by decision-makers concerned, parliamentarians and the general public in the MOPAN member countries;

! better informed dialogue with the multilateral organizations, both at headquarters and the country level; and

! improved overall performance of multilateral organizations at the country level.

Design Characteristics The design of the MOPAN Survey has two parts. The first part focuses on the quality of MO�s partnership behaviour towards national stakeholders (government, NGOs, private sector). The questions in this part of the survey solicit opinions on the following topics: policy dialogue, capacity development, advocacy, and alignment with national poverty reduction strategies, policies and procedures. The second part of the survey focuses on the quality of MO�s partnership behaviour towards other international development agencies. The questions in this part of the survey solicit opinions on the following topics: information sharing, interagency coordination and harmonisation.

Methodology The MOPAN Annual Survey is light and rapid with minimal transaction costs. It includes the filling-in by the participating MOPAN member embassies and country offices of a questionnaire on each of the multilateral organizations surveyed. Many MOPAN members work with the MOs through co-financing or participation in joint initiatives and donor meetings. They have an opportunity to directly observe their partnership behaviour. This is followed by joint discussions of questionnaire answers among MOPAN members at the country level (country teams). The joint group discussion provides a mechanism for testing individual views and forming a collective opinion. From these inputs, the country teams establish country reports, which are then aggregated into a Synthesis Report. This report is shared with the relevant multilateral organizations for feedback before its public release. The MOPAN Survey methodology has evolved over the years and now includes a series of tools and steps in what has become a well understood process:

• Agency template describes key aspects of each multilateral organization and includes information on mandate, structure and background. They are prepared by the MOPAN headquarter groups and provided to assist the country teams.

18 This is a summary of the essential components of the methodology adapted from CIDA MOPAN web site

and combined with the consultant�s interview notes. 19 The MOs selected for the 2006 MOPAN survey are UNICEF, ILO and ADB. The exercise took place in

the following countries: Burkina Faso, Kenya, Mozambique, Uganda, Indonesia, Nepal, Pakistan, Sri Lanka, Colombia, and Guatemala.

24

• Standard questionnaires are used to help each participating staff member to document his/her opinions on the multilateral organization�s performance. The questionnaire results are fed into the country and synthesis reports and are used as a basis for the group discussions by the MOPAN in-country teams.

• Country team discussions are used to share individual knowledge and views of the country team members and to �pool� the perceptions collected through the standard questionnaires. A collective overview of the organization emerges and the country team is able to expand their knowledge of the organization as well as develop a feeling of ownership of the results.

• The Country Report is a summary of the findings of the country team discussions. The report also includes information about the group process in reaching its consensus. The use of local consultants in the country report process is discouraged. This country report is generally discussed with the agencies concerned before finalization.

• The Synthesis Report is built up centrally by analyzing and summarizing the country reports for each multilateral organisation. The aggregate questionnaire data is analysed and used in the synthesis report.

• In an effort to promote transparency and dialogue among stakeholders, MOPAN has asked multilateral organizations to provide their feedback on survey conclusions, and post it on their websites together with the MOPAN Synthesis Report.

25

ANNEX C: SUMMARY OF THE MEFF METHODOLOGY20 Background In 2003-04, the DFID International Division (ID) established a Multilateral Effectiveness Framework (MEFF) for assessing the organisational effectiveness of the multilaterals that it funds centrally. It took 21 months to develop and administer the MEFF with twenty-three organisations, but the exercise has not been repeated since.

Purpose The main objectives of the MEFF are to:

! Provide an information and monitoring system that would support DFID�s reporting on its Public Service Agreement (PSA) objectives;

! Provide inputs to DFID�s corporate engagement with multilaterals via Institutional Strategies (ISs); and

! Provide inputs to future financing decisions.

Design Characteristics The MEFF design focuses on eight corporate management systems, i.e., corporate governance, corporate strategy, resource management, operational management, quality assurance, staff management, M&E lesson learning, and reporting, from three perspectives: internal performance, focus on country-level results, and partnership. It is designed to be applicable to all MOs, but recognizes the need to differentiate among different groupings: multilateral development banks, UN development organizations, UN standard-setting organizations, humanitarian organizations, coordinating organizations and the European Union Commission.

Methodology The MEFF uses three main assessment instruments, comprising a checklist of indicators, expressed as questions; a scorecard rating data in the checklists; and a summary report. These tools, associated processes and reports are described below: • The Checklist is the main assessment instrument which is designed as a matrix, with the eight

organisational systems listed horizontally and the three perspectives, vertically. This gives a total of 24 cells in the matrix, in each of which there are up to four questions.

THE MEFF CHECKLIST

Organizational systems Focus on Internal Performance

Focus on Country Level Results

Focus on Partnership

Corporate Governance 3 questions 3 questions 3 questions Corporate Strategy 3 questions 3 questions 3 questions Resource Management 4 questions 3 questions 2 questions Operational Management 4 questions 2 questions 3 questions Quality Assurance 4 questions 3 questions 2 questions Staff management 4 questions 2 questions 3 questions M&E Lesson learning 3 questions 3 questions 3 questions Reporting 3 questions 3 questions 3 questions

20 This is a summary of the essential components of the methodology adapted from �The MEFF

Methodology: A review of DFID�s multilateral effectiveness framework� by Allison Scott DFID, IDAD, March 2006 and combined with the consultant�s interview notes.

26

The questions in the Checklist avoided evaluative language that would require a subjective judgement on the part of the assessor. The questions were deliberately factual in nature: they asked whether a system or practice was in place or not or was under development. However, simple Yes or No questions were avoided. Guidelines were developed to provide key definitions and rules for completing the assessment. The answers were entered electronically into the checklist in the form of a short piece of text that supplied factual evidence in response to the question. Where possible, information sources were to be cited. The MEFF information base was thus textual and qualitative.

• The Scorecard summarised and illustrated the information generated from the checklists. It is

a simple traffic light system: green � the system or practice is in place; amber � it is under development; red � it is not in place. A blue score was entered if there was insufficient information available, and it was left blank if the question was not relevant to the agency. While the Scorecard colour system is criterion referenced to a certain extent, it nevertheless leaves considerable room for interpretation.

The Scorecard provided three levels of information. At the most disaggregated level (level 3), it contained the scores for each question in the matrix. At level 2, the scores were aggregated within each organisational system and by perspective, by registering the average or predominant score in the category, with a small chip to reflect the presence of higher or lower scores. At level 1 they were aggregated by perspective. The Guidelines provided rules for attributing scores and aggregating them to levels 2 and 1.

MULTILATERAL EFFECTIVENESS SCORECARD Dream Development Agency - Level 2 Summary

1 - Corporate Governance

2 - Corporate Strategy

3 - Resource Management 4 - Operational Management 5 - Quality Assurance

6 - Staff Quality 7 - Monitoring, Evaluation, 8 - Reporting

INTERNAL PERFORMANCE

COUNTRY-LEVELRESULTS FOCUS

PARTNERSHIPS FOCUS

2004 2005 2006 2004 2005 2006 2004 2005 2006

The system is in place

The system is not yet in place, but improvements are underway

No system in place yet

Predominant Colour at Level 2 Above predominant colour Below predominant colour

27

• Preparatory Meetings were held during the first three months with the MOs to explain the assessment process and to provide an opportunity for discussion of any questions or concerns. Each MO was provided with the full set of documents (approach paper, checklist and scorecard templates, guidelines).

• The First Draft checklist and scorecard were completed by the DFID desk officers. Thereafter

followed a period of consultation with the agencies on these drafts � more intensive in some cases than others. The MOs responded in various ways: they awaited the first draft and then commented; they implemented the checklist themselves with a view to comparing versions (e.g. WFP, AfDB); or they completed it jointly with DFID staff (e.g. IFAD, UNIFEM). In many cases, DFID lacked sufficient information to complete the checklist and had to request information from the MOs. Sometimes this took the form of them sending relevant reports, other times they suggested text for the checklist.

• While quality assurance was provided throughout the implementation process, the final adjustment to scores was minimal. The final step was to assess scoring consistency between assessments. DFID examined all the scores in the light of the evidence provided in the checklists and consistency of scoring across the 22 assessments. They found a small number of questionable scores (4.7% of all 1,574 scores), which were concentrated in only five MOs. Questionable scores mostly arose on the boundary between green and amber scores, usually because of a tendency to consider not just whether a system was in place, but whether it was working well. In the interest of accuracy and cross-agency consistency, the questionable scores were adjusted. This mainly had the effect of raising scores, but there were also several cases where they were lowered. Because of the small numbers involved, the overall impact of the adjustment was small both for individual MOs and in aggregate.

• The Final Dialogue Meeting was usually attended by senior management staff (Heads of Departments) of the MOs depending on how seriously they took the exercise. It involved detailed discussion of the textual answers as well as the scores. Although this was a consultation, not a negotiation, in most cases it was easy to reach agreement. Undoubtedly, some MOs were concerned to maximise the scores, but many were very self-critical and actually reduced DFID�s scores.

• The Summary Report provided an overview of the multilateral organisation�s strengths and weaknesses on each of the three perspectives, together with relevant background information on its structure and mandate, and any recent performance-related reforms. It also identified three areas that would be used for future monitoring of the organisation�s effectiveness.

28

ANNEX D: SUMMARY OF THE MES METHODOLOGY21

Background In 2006, DFID devised a �new approach� to assessing multilateral effectiveness. Rather than attempting to create, or recreate, a new all encompassing single tool to measure agency effectiveness, DFID has devised a new system which attempts to collate existing information from a variety of data sources and to present it within a coherent framework. This is the �Multilateral Effectiveness Summary�.

Purpose The MES will give users a general overview of an organisation�s effectiveness. It will include basic data on the organisation, the context in which the organisation is working, as well as a summary assessment of effectiveness. The summaries are aimed at providing senior managers and staff with a concise summary of all available information on the effectiveness of a particular multilateral organisation. While the information will be used to inform the management decision making process, it is not designed to produce a ranking of MOs by their effectiveness or to provide a basis for expenditure decisions. Rather the information will be used to help inform management decision making and to ensure senior managers have a consistent picture of each multilateral.

Design Characteristics In developing the MES, DFID has adapted the Balanced Scorecard (BSC) approach (see below), tailored it to meet the specific context in which multilateral organisations operate. The DFID BSC used to assess MO effectiveness focuses on four broad areas linked to the results chain.

Building for the Future - �How is the organisation building for the future through sharing information, learning and innovating?� This area considers whether the organisation is committed to a culture of continual learning and improvement, how it manages information and its results and whether it is investing in its staff.

Managing Resources � �How is the organisation managing its activities and processes?� This area looks at how the organisation uses its financial and human resources. It includes indicators such as disbursement ratios, resource allocation criteria, staff recruitment, postings and promotions processes and degree of decentralisation.

Partnerships - �How is the organisation engaging with other development partners?� This area looks at how the organisation works with other development partners including developing country governments themselves. It encompasses Paris Declaration Indicators on alignment and harmonisation.

Country/Global Performance - �How well is the organisation performing either at the country level and/or at the global level?� The final area considers any information which is available on actual performance either at the country level or at the global level. This includes results of any evaluations, by the organisation themselves or by other organisations, information on portfolio quality, any data on programme targets and any other indicators on outcomes and impact.

These four areas represent the quadrants of the BSC and can be seen as broadly following the results chain from inputs to outputs to outcomes and impact. They focus on the MOs institutional effectiveness and provide proxy measures of developmental effectiveness.

21 This is a summary of the essential components of the methodology adapted from two DFID Practice papers

�Factsheet� (June 2006) and �Methodology Note� (November 2006) and combined with the consultant�s interview notes.

29

Methodology While the conceptual framework and the tools designed for the MES are well documented, the data collection and analysis processes are less well described. It appears that the MES will be completed by DFIS staff using secondary data from a variety of sources, including: the MEFF, the MOPAN Annual Survey, the Paris Declaration indicators, as well as the MOs own performance reports. Below are the key tools and reports that would be used to support the MES preparation process. • The Balanced Scorecard that DFID has adapted reflects its own business needs and is used

for its own Quarterly Management Report. DFID proposes to use essentially the same framework as presented below to assess the effectiveness of multilateral organizations.

DFID Quarterly Management Board Report Framework

• A Traffic Light Assessment will be made in the future based on the information collated for each of the above four quadrants. As far as possible, they would be based on objective evidence from the published data sources. However, in cases where the evidence base is weak, judgements of desk officers working with the particular multilateral organisations will also inform the overall assessment.

• The Multilateral Effectiveness Summaries are short publications which will have a

standardised format as follows: MO tombstone data, the BSC/Traffic Lights Assessment, a Disclaimer (below), the findings on the four areas of the BSC and data references.

DFID Mission: Poverty reduction

through partnerships

Managing Resources�How well are we planning

and managing our resources?�

Managing External Relationships

�Are we managing our relationships effectively?�

Building for the Future�Are we delivering our people

and organisation for the future?�

Delivery of Results�Are we delivering against

our PSA targets?�

*Disclaimer This Effectiveness Summary is a tool designed to simply present the latest available published information on multilateral X�s effectiveness. It will help inform our policy decisions but will not be used as the only basis to make policy recommendations on funding. It is intended for internal use only. Although the balanced scorecard assessment aims to be as objective as possible, the colours reflect our judgement of what we think the information tells us. It should be noted that the amount of information available and the quality and reliability of that information varies considerably, so there is a limit to which we can use the summaries for comparative purposes.

30

ANNEX E: SUMMARY OF THE MERA METHODOLOGY Background The Canadian International Development Agency (CIDA) has developed a standard template of questions called the Multilateral Effectiveness and Relevance Assessment (MERA) system. It is relatively new having been used but once in 2006.

Purpose The main objectives of the MERA are:

! To better inform policy and financial allocation decisions on more solid evidence. Ultimately, we aim at getting more predictable multi-year funding, and an adequate balance between core and responsive/earmarked funding;

! To better exercise our accountability and improve our reporting to Canadian citizens;

! For more effective Board meetings and better identification of areas requiring improvements; and

! To deliver on MPB commitment to complete the multilateral effectiveness review, as part of the Agency Aid Effectiveness agenda.

The findings will be used as a basis for making evidence-based policy decisions and financial allocations. They will help identify areas in multilateral organizations requiring improvement. They will also improve Branch accountability, the effectiveness of our Board interventions, and reporting to Canadians. This should eventually allow CIDA to link funding decisions to the effectiveness of the organizations.

Design Characteristics The MERA provides a framework for comparing multilateral organisations receiving core funding from Multilateral Programs Branch, CIDA. Its design is organised around three main criteria: relevance, effectiveness, and management. Each MO is reviewed through a set of questions covering these three criteria. The �relevance� criteria include questions regarding the MOs strategic coherence with the MDGs, Canadian development priorities, and its role within the multilateral aid architecture. The �effectiveness� criteria include questions regarding the MOs own performance rating, achievement of sustainable results, and the effectiveness of its advocacy and norm setting work. The �managing of the institution� criteria include questions regarding governance and financial management, ability to manage for development results, compliance with aid effectiveness principles, and ability to mainstream gender equality and environment considerations. The majority of the questions are evaluative in nature requiring the assessor to make an informed judgement based on the available information. MOs are rated on a normative five point numeric scale on each of the three criteria.

Methodology Assessments are conducted by the responsible MPB Desk Officers who draw on any multiple data sources: two surveys (MOPAN and field survey on multilateral institutions), reporting from multilateral organizations and their evaluations, external evaluations commissioned by CIDA or conducted jointly with other donors and due diligence. Individual assessments are reviewed and validated by the Branch Management Group to mitigate any extreme opinions. The process is admittedly subjective using mostly evaluative questions with some empirical questions. The next step is said to review the current level of funding to these multilateral organisations and propose a new allocations, as required, to address weaknesses. The following standard template is applicable to all multilateral organisations.

31

Source of information

Relevance Effectiveness Improvement measures (by the institution and by MPB)

Links to MDGs: # General link between

MO�s mandate and MDGs; # Specific MDGs targeted

by MO

Overall governance # General overview of MO; # Donor or recipient driven; # Resident board or executive

council # Independent evaluators�

impression of governance; # General reputation; # Personal insights of evaluator

# Status of donor or MO

driven specific management improvements (e.g. implementing a report, instituting a new policy, efforts to improve RBM)

# General improvements (e.g. ensuring strong voice on Board on key issues)

Role in the Multilateral architecture: # How MO fits in multi

development system Region /sector specific, UN or IFI or other

# Kind of aid loans, grants # Budget support # Tech assistance # Partnership with another

MDI # role within the MDI

system in a particular area

Efficient use of resources # Operating costs as a percent of

its aid delivery # Strengths and weaknesses,

factors which hinder efficiency

Support to IPS sectors: # Sectoral breakdown of

its most recent year of programming

Donor harmonization and alignment � Paris Declaration # Work towards objectives of the

Paris declaration # Harmonizing/aligning with

other UN orgs or IFIs and bilaterals

Managing for results # Extent of RBM systems # Reporting systems

Quality of reporting, monitoring & evaluation systems

Gender mainstreaming: # Policy # Implementation

CIDA Desk Officers can draw from:

MOPAN surveys

Other donor

surveys

CIDA field surveys

Reports from the MO itself

Evaluation reports

Environment mainstreaming: # Policy # Implementation

Overall assessment: Description of how relevant and effective the institution is both in its own right and compared with other MDIs and donors comes in.

32

ANNEX F: SUMMARY OF THE SCORECARD METHODOLOGY22 Background The Netherlands Ministry of Foreign Affairs (MFA) has been conducting annual performance assessments for the most important MOs at the country level of the 36 Dutch partner countries since the Beoordelingskader (BOK) questionnaires of 2002 and 2003 sent to embassy staff, permanent mission representatives and constituency offices staff. In 2003 BOK was replaced by MMS (Multilateral Monitoring System) and in 2004 the scorecard method was introduced. This Scorecard method is considerably more detailed and is more robust than the way MOs were assessed in the past. While the Scorecard has been an internal document to the Dutch MFA for sometime, it has recently been used more openly.

Purpose The Scorecard plays an important role in informing the MFA�s policy towards the MOs concerned, including in general policy dialogue, interventions at governing council meetings, and may also form the basis for professionalizing financial relationships with the MOs by influencing the amount and nature of its voluntary contributions. It is not used to draw firm and unequivocal conclusions, because there is a subjective element in the weighting of criteria. In addition, considerations other than MO effectiveness also play a part in decision-making on how Dutch resources are allocated, such as the perceived role of the MOs in the multilateral system.

Design Characteristics Drawing up the Scorecards is not a one-off exercise and must be interpreted from a multi-year perspective. It reflects the quality of the MO�s performance over time, the extent to which it has adopted a results-based approach and how relevant its programming has been to the Netherlands and to Dutch policy objectives. Three assessment criteria are used which focus on: a) internal performance, such as results-based management (RBM), financial management and regularity, and anti-corruption activity; b) external performance, such as focus on core mandate, coordination/cooperation; and c) relevance to international development cooperation priorities, i.e. MDGs. A five point normative Lickert scale is used to rate MO performance and relevance. Various data sources are used in completing the Scorecard, including experiences at MFA headquarters and other Dutch Ministries, the perceptions of policy departments, data collected from the MMS and MOPAN surveys, other findings from the field and the MO�s own evaluation reports.

Methodology While the Scorecard template remains available only in Dutch and the methodology for completing it remains undocumented and somewhat unfamiliar to most, the MMS is much better known which may have lead to the misconception that the latter constitutes the Dutch MO assessment process. What is known is that the Scorecard assessments are conducted by the responsible MFA Policy Officer who solicits the opinions of informed staff in other Ministries and draws on the available data generated from the MMS and the MOPAN surveys, as well as content analysis of MO strategy documents, evaluation reports and performance reports, etc.

The MMS is a perception survey of Embassy staff who is asked to update the previous performance assessment of the MOs active in their respective countries. For methodological uniformity, Embassy staff describes and rate the MOs based on the same criteria as used in the Scorecard. They rate their level of satisfaction on fifteen (15) questions using the following

22 This is a summary of the essential components of the methodology adapted from the �Fact Sheet for

Multilateral Monitoring� (April 2007) and combined with the consultant�s interview notes.

33

normative rating scale: + satisfactory, - unsatisfactory, and ? don�t know. The perceptions gained are subjective, but taken together they give an impression of how the MOs perform at a country level. So, instead of objectivity and automatic conclusions, the MMS provides collective subjectivity. Since perceptions don�t appear to change significantly over a year, the MMS is updated every other year (biannually) which makes it possible to identify trends in the MO�s performance, both nationally and internationally. The regional departments and the United Nations and International Financial Institutions Department (DVF) check the MMS data for consistency. In case of different views the mission is contacted and asked for additional information. DVF places the results on the intranet, so that they are available to everyone at the Ministry. The 2005 MMS round at country level has been completed and can be found on the MFA intranet site at country level. The next MMS round has taken place in 2007. Results are being received and processed at the moment. The updated MMS data is integrated into the Scorecard Report. A recently updated Scorecard Report contains the following headings and information:

Netherlands Scorecard

GENERAL BACKGROUND Aim and mandate (narrative description) Working method and activities (narrative description) Financial scope and contributions (narrative description) Dutch representation (narrative description) Trends among other donors (narrative description) PERFORMANCE REVIEW Cluster 1: Internal performance

1. Effectiveness of administrative and management (rated assessment and description) 2. Transparency (rated assessment and description) 3. Monitoring and evaluation (rated assessment and description) 4. Results-based management (rated assessment and description) 5. Financial management and legitimacy (rated assessment and description)

Cluster 2: External performance 1. Focus on core mandate (rated assessment and description) 2. Coordination and cooperation with multilaterals (rated assessment and description) 3. Coordination and cooperation with bilaterals (rated assessment and description) 4. Implementation of policy at country level (rated assessment and description)

Cluster 3: Relevance A. General relevance of the institution (rated assessment and description) B. Specific relevance for the priorities in Mutual interests, mutual responsibilities

1. MDGs a) General (rated assessment and description) b) HIV/AIDS (rated assessment and description) c) Reproductive health (rated assessment and description) d) Basic education (rated assessment and description) e) Water and sanitation (rated assessment and description) f) Environment (rated assessment and description) g) Gender (rated assessment and description)

2. Security and stability (rated assessment and description) 3. Private sector development (rated assessment and description) 4. Africa (rated assessment and description) 5. Good governance (rated assessment and description) 6. Regional approach (rated assessment and description) 7. Coherence, trade and debt policy (rated assessment and description)

CONCLUSION (narrative description)

34

ANNEX G: SUMMARY OF THE PMF METHODOLOGY23 Background In 2003, Danida experienced a corporate restructuring which included the decentralisation of responsibilities for bilateral and multilateral programming and monitoring to the field. While the Copenhagen Headquarters retains responsibility for policy development and guidance, the permanent missions cover the MO Board Meetings. The development of the Performance Management Framework (PMF) by Danida HQ was in response to this context of an increasingly decentralized management structure and the subsequent need for a performance monitoring and management tool. Since 2004 assessments of MO effectiveness have been conducted on an annual basis.

Purpose The objectives of PMF are to:

! enhance the quality of Danish development cooperation through stronger focus on results;

! improve management and continuous learning, through better information and reporting; and

! strengthen accountability through performance assessments and measurement in the context of an increasingly decentralized management structure.

The PMF systematically focuses on results with a view to providing performance information for management decisions that optimises value for money and ensures the prudent use of human and financial resources in support of the overall objective of poverty reduction. The performance information creates a basis for timely and relevant decisions and/or corrective action at all levels in Danida and hence improves the internal management and the decision-making process. By encouraging the formulation of clearer objectives, better documentation of results and systematic monitoring of progress towards achieving these, the PMF helps Danida and partners to strengthen the effectiveness of policies and programmes.

Design Characteristics The PMF is not an assessment tool per se, but rather a corporate framework and a set of specific tools for managing performance. The framework has two major parts for: Bilateral Development Cooperation and Multilateral Development Cooperation. The PMF for multilateral cooperation has three levels of analysis: 1) Danida Corporate level, 2) MO HQ level, and 3) MO Country level. At the Danida Corporate Level, the PMF enables Danida to assess:

1. Overall progress towards reaching the aim of poverty reduction through the multilateral organisations.

2. The multilateral organisations� contribution towards objectives regarding cross-cutting issues of gender equality, environmental concerns, as well as promotion of human rights, democracy and good governance.

3. The multilateral organisations� contribution towards improving aid effectiveness, including support for processes of alignment and poverty

23 This is a summary of the essential components of the methodology adapted from �Performance

Management Framework for Danish Development Cooperation 2006-07� prepared by the Quality Assurance Department, KVA (December 2005) and combined with the consultant�s interview notes.

35

reduction strategies in partner countries, as well as progress in donor coordination and harmonization.

At the MO HQ Level, the PMF is based on the �organisation strategies� for each of the major multilateral organisations. They are formulated by Danish multilateral Representatives and relevant departments in the MFA in very close consultation with the MO, and define the strategic priorities for the Danish cooperation with the organisation. The agreed to objectives and priorities are to be taken from the strategic document of the MO to ensure full alignment and to facilitate subsequent reporting. While organisation strategies cover a three-four year period, annual action plans define indicators and targets to be achieved during each year. Danida draws on the information generated in the context of high-level consultations and by the assessment forms filled out by multilateral Representations and relevant departments of the MFA. Also, in line with international commitments to reducing the transaction costs of development cooperation, Danida is interested in relying more on performance reporting provided by the multilateral agencies themselves. However, it is a precondition for this approach that reliable information is provided by the MOs themselves. The Danida RBM Assessment program is an initiative that addresses the current weaknesses in MO reporting on development results.

At the Country Level, the PMF may vary from one country to another, and various systems to assess the country-level performance are established both by the MOs themselves and by the bilateral donors. Danida participates together with like-minded donors in the joint donor assessments of multilateral organisations� effort at the country level (MOPAN), which usually focus on three organisations in 8-10 countries every year. To supplement this information, Embassies in programme countries draw up annual assessments of the field-level performance of the three (3) most important MO-partners operating in the country.

Methodology The PFM includes a series of tools for its quality assessment of MOs, including: Organisation strategies govern the Danish cooperation with the multilateral organisations receiving more than DKK 20 million annually as well as organisations that are strategically important to Danish multilateral cooperation. These strategies outline the overall Danish priorities vis-à-vis the organisation and progress in meeting these objectives is measured annually through action plans which contain indicators of performance in pre-determined areas. High-level Consultations between the MFA and each of the multilateral organisations take place at least biannually to assess the organisation�s progress towards implementing the Danish strategy for the organisation and issues relating to the organisation�s handling of all general and earmarked contributions provided by Denmark. The minutes of these meetings constitute important monitoring information. Perception analyses of the general performance of multilateral organisations are made by multilateral representations and relevant departments in MFA. Joint donor assessments of multilateral organisations� effort at country level (MOPAN) are carried out annually in a joint effort between Denmark and like-minded donors. Perception analyses of the performance of multilateral organisations at country level are carried out by Embassies in programme countries.

36

Evaluations/assessments of multilateral organisations� performance reporting systems are carried out jointly with other donors to provide the basis for support to strengthening the capacity of the organisations to provide high-quality performance information themselves.

Overview of the PMF for Multilateral Development Cooperation Focus of the PMF Key Management

Documents Methodology &

Data Sources Reporting and Responsibilities

Danida Corporate Level

Measure performance and verify results on overall goals, objectives and indicators in key policy documents.

- Partnership 2000

- MDGs

- Various sector or issue-specific policies and strategies

- Annual Report on Action Plans

- MO Assessments (HQ & Field)

- Evaluations

- MOPAN Survey

- Annual Performance Report (KVA)

- Annual Report of Danida (UDV)

MO HQ Level

Monitor performance against the objectives, targets and indicators set out in the MO�s own vision & strategy.

Monitor performance against the objectives, targets and indicators set out in Danida�s Organisation Strategies. (subset of above).

- MO strategies and action plans

- Organisation Strategies and action plans (Danida�s)

- Multilateral and relevant department�s MO Assessments

- RBM Assessment

- Evaluations

- Minutes from High level Consultations and annual Report on action Plans


- Report to the Board of Danida

- RBM Assessment Report (KVA)

- Evaluation Reports (EVAL)

Country Level

Monitor performance at country level against the objectives, targets and indicators, set out in the MO�s own vision and strategy.

- MO strategies and action plans

-Embassy staff assessment of MOs

- MOPAN Survey


- Assessment Reports (KVA)

38

ANNEX H: SUMMARY OF THE RBM ASSESSMENT METHODOLOGY24 Background In line with international commitments to reduce the transaction costs of development cooperation, Danida was interested in relying more on performance reporting provided by the MOs themselves. Consequently, the Danida PMF included among its various data collection techniques Evaluations/assessments of multilateral organisations� performance reporting systems in order to determine the robustness of the MOs performance reporting systems, but also to strengthen their capacity to provide high-quality performance information. In 2005, the consulting firm Dalberg � Global Development Advisors was contracted to undertake a number of RBM Assessments of a select number of UN Agencies, i.e., UNDP, UNFPA and UNICEF.

Purpose The goals of the RBM assessment are to map, analyse and provide recommendations for the strengthening of an MO�s RBM system. In short, the assessment should seek to identify both what RBM tools are in place as well as how effectively they are used in pursuit of improved organizational effectiveness and efficiency at delivering development results.

Design Characteristics The RBM Assessment is designed to examine three key elements in an RBM system as described in the conceptual framework below.

The assertion is that all three of the above elements need to be sufficiently robust for the organization to have a results-based management culture. A robust RBM architecture replete with logic models, strategic outcomes, indicators, data collection tools and information management systems is necessary but insufficient without the human processes that facilitate data collection, analysis, usage of findings for management decision making and program delivery adjustments. These human processes however remain generally weak and ineffective without staff incentives and disincentives to make the RBM system work, as well as the public demand by senior management for credible and reliable performance information to consider when making resource allocation decisions. 24 This is a summary of the essential components of the methodology adapted from �Expanding the RBM

Assessment Program� prepared by the Netherlands MFA (June 2006) and combined with the consultant�s interview findings.

39

Methodology The methodology of conducting an RBM assessment must go beyond a systems theory approach and examine the corporate management culture to determine if it is sufficiently supportive of managing for results. Assessing the three key elements in the RBM system is nevertheless a good first step as described in the table below.

However, the methodology must also ensure senior management buy-in and engage the MO in a participatory process during the subsequent steps depicted in the diagram below.

40

ANNEX I: SUMMARY OF THE FARP METHODOLOGY25 Background A framework for assessing relevance and performance of multilateral organisations was designed and launched in early 2007 by the Swedish Ministry of Foreign Affairs in response to demands for improved strategic action, better results orientation and greater monitoring and accountability for multilateral effectiveness. The need for such a framework and the main criteria therein were spelled out in the Strategy for Multilateral Development Cooperation, which was adopted by the Government on April 3, 2007. The framework was developed in order to be able to feed into the budget process for 2008. Central to the assessment framework is an assessment template (scorecard) which was developed under heavy time-pressure and without sufficient testing. It will, thus, be subject to further refinements for next year�s budget process. Twenty-eight (28) MOs (UN organizations, development banks, EU and vertical funds) were assessed during this first cycle.

Purpose The annual structured assessment procedure serves two main purposes, namely to:

• guide decisions on allocations of funds to MOs,

• provide an analytical basis for strategic dialogue with the MOs, and provide the basis for developing specific organizational strategies for those MOs.

The assessments will also feed in to the Government�s annual reporting on development cooperation to the Swedish Parliament.

Design Characteristics The template for assessing MOs receiving Swedish funding is designed in a qualitative and structured format. The principal criteria in the assessment template are relevance and effectiveness. Relevance means the compatibility of the activities with Swedish development goals and the role of the organisation in the international aid architecture. Effectiveness means whether the organisation contributes to the relevant goals set and whether the activities are organised so as to lead to results and employ aid resources effectively (i.e. internal and external effectiveness). The idea is to provide a snap-shot of where the organization is at and where it is moving. There is, however, a special section, allowing for comments on reform processes and positive/negative trends.

The first section, on relevance of goals, is structured along the lines of priorities established in Sweden�s Policy for Global Development and allows for four categories of response (concerning degree of compatibility, from �very high� to �none�).

The other sections allow for only three categories of response (�yes�, �no�, �unable to assess�). For each of the four sections an assessment is made in terms of �very relevant/effective�, �relevant/effective�, or �not relevant/effective�. Limited space is provided to comment the assessment made. The second section concerns how the organization fits into the international aid architecture. The third examines the internal effectiveness by asking questions about the organizational structure, RBM focus, existing monitoring and evaluation systems, and transparency/disclosure policy, including reporting on auditing. The fourth, examines external effectiveness by asking questions on selectivity, cooperation with others (i.e. in line with the Paris 25 This is a summary of the essential components of the methodology based on a recent unofficial translation

of the country level questionnaire and combined with the consultant�s interview findings.

41

declaration), and the degree of focus at country level. Finally, there is a fifth section that allows for comments on planned or on-going reforms and other expected changes (or trends).

As an annex to the assessment, staff filling in the template have been asked to summarize the most important findings from other existing evaluations and the like, of which the analysis is based upon. Examples of such source of information have been internal evaluations of the organization concerned, MOPAN studies, and evaluations done by Sida and other development partners. A special template was also designed and sent out to 20 embassies ahead of the overall assessment in order to get a perspective from the field. This template poised questions regarding 11 different MOs. A Lichert scale methodology was used � i.e. the questions were actually statements to which one could indicate, along a scale of five, the degree of compatibility.

A second annex to the assessment provides statistical data about the total size of resources the organization has disbursed over the last five year period. Sweden�s contribution over the same period, Sweden�s ranking as donor as well as total donor contributions were also included.

Methodology A special project group within the Ministry of Foreign affairs has been responsible for developing the assessment template. The group also developed the special template sent out to 20 selected embassies prior to the overall assessment period, as it could serve as an input. Given expectations from the political leadership to have something to show already in the budget process for 2008, the templates were developed without a sufficient quality assurance process. The current template is thus a bit of an experiment, from which there will many lessons learned to be drawn. The overall assessment templates was filled in through a consultative process, lead by the responsible desk officer at the responsible Ministry in Stockholm. Input received from the template sent to the embassies was collected by the project group and put together by organization, as a service to the team. The team leader�s task was then to coordinate the job of filling in the assessment with his/her peers working specifically with the organization. That meant staff at Sida, Sweden�s UN Representations and other Ministries. In theory there were four weeks to undertake the assignment in practice, given Easter and other holidays, on top of the regular work-load, it was much less. The completed assessments were sent to the project group for quality assurance. Three aspects were considered particularly important � completeness, internal consistency between comments and ratings, and to what degree overall assessments seemed honest enough. The project group, together with division heads went through all assessments to examine these aspects. Although, the quality of assessments were of much better quality than anticipated, a number of team leaders were asked to take a second look at their assessments. The final assessments were then sent to a special reference group consisting of heads of departments to discuss if the assessments were of sufficiently good quality and how they could be used in this year�s national budget allocation process. This is where the process stands today! Given the methodological weaknesses and novelty of the whole process, it is probably safe to say that the assessments will mainly serve as a guideline for dialogue with the MOs and to a lesser extent as an impact on budget allocations.

42

AN

NEX

J: C

RO

SS-C

ASE

CO

MPA

RIS

ON

TA

BLE

OF

MU

LTIL

ATER

AL A

SSES

SMEN

T AP

PRO

ACH

ES

NA

ME

M

EFF

M

ES

� B

SC

ME

RA

Sc

orec

ard

PMF

RB

M

Ass

essm

ent

FAR

P

Bila

tera

l Don

or

DFI

D

DFI

D

CID

A

MFA

- N

ethe

rland

s D

AN

IDA

D

AN

IDA

M

FA -

Swed

en

Bac

kgro

und

and

Purp

ose

Ince

ptio

n Y

ear

2004

20

07

2006

20

04

2003

20

06

2007

Fr

eque

ncy

Onc

e O

nce

Onc

e A

nnua

l A

nnua

l U

pon

Requ

est

Onc

e #

of M

Os (

/ yea

r)

23

N/A

19

25

MO

s / y

ear

3

MO

s/yr

/cou

ntry

ac

ross

15

coun

tries

U

ND

P, U

NFP

A,

IFA

D, O

HC

HR

, U

NIC

EF

28 M

Os /

yea

r

Obj

ectiv

es

Prov

ide

info

rmat

ion

for a

genc

y�s:

(a

) Pub

lic S

ervi

ce

Agr

eem

ent

repo

rting

, (b)

its

Inst

itutio

nal

Stra

tegi

es, (

c) it

s fin

anci

ng st

rate

gies

To e

nsur

e se

nior

m

anag

ers h

ave

a co

nsis

tent

pic

ture

of

the

effe

ctiv

enes

s of

eac

h M

O a

nd to

he

lp in

form

m

anag

emen

t de

cisi

on m

akin

g.

To d

eter

min

e ef

fect

iven

ess a

nd

rele

vanc

e w

ith a

vi

ew to

dire

ctin

g fu

ture

reso

urce

s to

enha

nce

MO

pe

rform

ance

.

To a

sses

s the

re

leva

nce

of M

Os

to D

utch

pol

icy

obje

ctiv

es, t

heir

perfo

rman

ce a

nd

adop

tion

of R

BM.

To:

(a) e

nhan

ce th

e qu

ality

of d

evel

op-

men

t co

oper

atio

n,

(b) i

mpr

ove

man

age-

men

t and

co

ntin

uous

le

arni

ng, (

c) st

reng

- th

en a

ccou

ntab

ility

Prov

ide

supp

ort t

o:

(a) s

treng

then

M

Os R

BM

ca

paci

ty (b

) en

ablin

g bi

late

ral

dono

rs to

dep

end

on M

O re

sults

re

porti

ng fo

r as

sess

men

ts.

To i

nfor

m b

udge

t de

cisi

on-m

akin

g pr

oces

s, an

d to

pr

ovid

e an

an

alyt

ical

bas

is fo

r st

rate

gic

wor

k w

ith

the

MO

s.

Des

ign

Cha

ract

eris

tics

Ass

essm

ent

Crit

eria

and

/or

Pers

pect

ives

Eigh

t or

gani

zatio

nal

syst

ems a

nd th

eir

focu

s on

(a) i

nter

nal

perfo

rman

ce,

(b) c

ount

ry le

vel

resu

lts a

nd

(c) p

artn

ersh

ips

Four

qua

dran

ts o

f th

e B

alan

ced

Scor

ecar

d: a

) B

uild

ing

for t

he

Futu

re, b

) M

anag

ing

Res

ourc

es, c

) Pa

rtner

ship

s and

d)

Cou

ntry

/Glo

bal

Perfo

rman

ce.

(a) r

elev

ance

, (b

) res

ults

(c

) man

agin

g of

the

inst

itutio

n

Scor

ecar

d cr

iteria

: a)

inte

rnal

pe

rform

ance

, b)

ext

erna

l pe

rform

ance

, and

c)

rele

vanc

e

(a) D

anid

a C

orpo

rate

leve

l;

(b) M

O H

Q le

vel;

(c

) Cou

ntry

leve

l (fi

eld

leve

l)

RB

M S

yste

m:

a) A

rchi

tect

ure

b)Pr

oces

s, an

d

c) In

cent

ives

/ R

esou

rce

Allo

catio

ns

Rel

evan

ce: a

) goa

l co

ngru

ence

and

b)

role

rele

vanc

e A

genc

y

perfo

rman

ce c

) in

tern

al p

erfo

rm.,

and

d) e

xter

nal

perfo

rman

ce

# of

Crit

eria

8

4 3

3 3

3 4

Inte

rnat

iona

l and

N

atio

nal p

riorit

ies

Intn

�l:

MD

Gs

and

Paris

D

ecla

ratio

n:

Intn

�l:

MD

Gs

and

Paris

D

ecla

ratio

n:

Intn

�l:

MD

Gs

and

Paris

D

ecla

ratio

n In

tn�l:

M

DG

s an

d Pa

ris

Dec

lara

tion;

In

tn�l:

M

DG

s, PR

SPs

and

cros

s-N

/A

Intn

�l:

MD

Gs

and

Paris

D

ecla

ratio

n;

43

AN

NEX

J: C

RO

SS-C

ASE

CO

MPA

RIS

ON

TA

BLE

OF

MU

LTIL

ATER

AL A

SSES

SMEN

T AP

PRO

ACH

ES

NA

ME

M

EFF

M

ES

� B

SC

ME

RA

Sc

orec

ard

PMF

RB

M

Ass

essm

ent

FAR

P

addr

esse

d pa

rtner

ship

an

d ha

rmon

isat

ion

partn

ersh

ip

and

harm

onis

atio

n C

anad

ian

prio

ritie

s:

GE,

EN

VIR

, an

d R

BM

Dut

ch

prio

ritie

s:

e.g.

, PS

D,

GG

DD

, an

ti-co

rrup

tion,

etc

.

cutti

ng i

ssue

s, e.

g.,

GE,

H

RD

GG

, EN

VIR

Swed

ish

prio

ritie

s:

HR

GG

DD

,GE,

NR

, ec

onom

ic

grow

th,

soci

al d

evel

opm

ent

and

secu

rity,

et

c.

M

etho

dolo

gy

Ove

rall

Met

hodo

logi

cal

App

roac

h

Ass

essm

ent i

nitia

lly

cond

ucte

d by

DFI

D

staf

f w

ith

vario

us

degr

ees

of

inpu

t fro

m

MO

s an

d en

ding

w

ith

final

di

alog

ue m

eetin

g.

Ass

essm

ent i

nitia

lly

cond

ucte

d by

DFI

D

staf

f us

ing

vario

us

seco

ndar

y da

ta

sour

ces.

Ass

essm

ent

cond

ucte

d by

CID

A

MPB

D

irect

orat

e st

aff

draw

ing

on

pers

onal

per

cept

ions

an

d av

aila

ble

empi

rical

dat

a.

Scor

ecar

d up

date

d by

M

FA

Polic

y O

ffice

r he

avily

ba

sed

on

opin

ion

surv

eys

and

othe

r av

aila

ble

empi

rical

da

ta.

Ass

essm

ent l

ead

by

MFA

Des

k O

ffice

r us

ing

mul

tiple

line

s of

ev

iden

ce,

incl

udin

g Em

bass

y st

aff

surv

ey

and

RB

M a

sses

smen

t

Ass

essm

ent o

f MO

s R

BM

sy

stem

s in

H

Q/fi

eld

cond

ucte

d by

con

sulta

nts

in a

hi

ghly

col

labo

rativ

e m

anne

r.

Ass

essm

ent b

y MFA

D

esk

Offi

cer

who

le

ads

the

team

dr

awin

g on

th

e in

form

ed o

pini

ons o

f e.

g.

cons

titue

ncy

offic

es, D

eleg

atio

ns

and

field

staf

f. D

ata

sour

ces

DFI

D

staf

f w

ith

vario

us d

egre

es o

f in

put

from

M

Os

staf

f and

man

ager

s

Seco

ndar

y da

ta

sour

ces:

M

EFF,

M

OPA

N

Surv

ey,

Paris

Sur

vey,

DA

C

peer

rev

iew

s, M

O

perfo

rman

ce

and

eval

uatio

n re

ports

.

CID

A

staf

f, av

aila

ble

docu

men

tatio

n an

d M

O

perfo

rman

ce

repo

rts.

Scor

ecar

d so

urce

s:

HQ

ex

perie

nces

, po

licy

depa

rtmen

t pe

rcep

tions

, M

MS

Surv

ey w

ith 1

5 Q

s, M

OPA

N

Surv

ey,

othe

r fin

ding

s fro

m

the f

ield

and

anal

ysis

of M

O e

valu

atio

ns.

Mul

tiple

so

urce

s:

MFA

Des

k O

ffice

rs,

Reg

iona

l D

eleg

atio

ns

(NY

, G

EN,

WA

SH,

RO

ME)

, Em

bass

y st

aff,

MO

sta

ff an

d m

anag

ers

(RBM

A

sses

smen

t)

Mul

tiple

so

urce

s:

MO

doc

umen

tatio

n,

staf

f int

ervi

ews,

on-

site

ob

serv

atio

ns,

test

ing

or

view

ing

dem

onst

ratio

n of

on

line

syst

ems

/repo

rting

. G

roup

di

scus

sion

s an

d w

orks

hops

with

MO

m

anag

emen

t & st

aff.

Man

dato

ry:

Surv

ey

of

20

Emba

ssie

s;

Del

egat

ion

HQ

in

put;

SID

A

them

atic

in

put;

anal

ysis

of

au

dit

repo

rts a

nd a

nnua

l re

ports

. A

dditi

onal

: M

OPA

N

repo

rts;

MO

ev

alua

tions

, pe

er

revi

ews

asse

ssm

ents

by o

ther

do

nors

. #

of K

PI/Q

s 72

40

26

15

21

33

33

Type

of K

PI/Q

s Em

piric

al

Mix

ed

Eval

uativ

e M

ostly

eva

luat

ive

Mos

tly e

valu

ativ

e M

ixed

Ev

alua

tive

Rat

ing

scal

es o

r Y

es �

3 p

oint

/col

our

Yes

-Bei

ng

Yes

�

norm

ativ

e Y

es

� Sc

orec

ard

Yes

�

Emba

ssy

No

Yes

� 3

poi

nt ra

nk-

44

AN

NEX

J: C

RO

SS-C

ASE

CO

MPA

RIS

ON

TA

BLE

OF

MU

LTIL

ATER

AL A

SSES

SMEN

T AP

PRO

ACH

ES

NA

ME

M

EFF

M

ES

� B

SC

ME

RA

Sc

orec

ard

PMF

RB

M

Ass

essm

ent

FAR

P

tool

s use

d T

raffi

c Li

ghts

A

sses

smen

t de

velo

ped

B

SC

with

5 p

oint

/ col

our

Traf

fic

Ligh

ts

Ass

essm

ent,

but

won

�t be

ap

plie

d un

til 2

008.

num

eric

1-

5 fo

r ea

ch o

f 3 c

riter

ia

uses

nor

mat

ive

4-5

poin

t ra

nk-o

rder

ed

scal

es

MM

S R

atin

g sc

ale:

+

satis

fact

ory

- uns

atis

fact

ory

? D

on�t

know

surv

ey

uses

no

rmat

ive

rank

-or

dere

d 5

poin

t sa

tisfa

ctio

n sc

ale:

V

S,S,

US,

VU

S

orde

red

scal

e to

rate

re

leva

nce

and

effe

ctiv

enes

s. Fo

r ea

ch s

ub-in

dica

tor:

gene

rally

3 o

ptio

ns:

�yes

�,

�no�

or

�u

nabl

e to

ass

ess�

.

Scor

es u

sed

to

com

pare

Y

es -

Colo

ured

D

ashb

oard

N

o Y

es

No

No

No

No.

Ince

ntiv

es a

nd R

esou

rce

Allo

catio

ns

Rep

ort s

hare

d w

ith M

Os i

n co

untry

Yes

N

/A

No

No

No

Yes

N

o, b

ut w

ill p

rovi

de

feed

back

at

hi

gh

leve

l con

sulta

tions

Su

mm

ary

Rep

ort

publ

icly

ava

ilabl

e Y

es

Effe

ctiv

enes

s Su

mm

arie

s fo

r 16

M

Os

will

be

pu

blis

hed

in

July

20

07.

No

Sum

mar

y re

port

on

each

MO

bas

ed o

n M

MS

data

is g

iven

to

Po

licy

Offi

cer

whi

ch

is

then

in

putte

d in

to

the

Scor

ecar

d w

hich

has

not

been

a p

ublic

do

cum

ent.

Yes

Y

es

Not

yet

dec

ided

for

th

e fu

ture

� f

or t

he

first

yea

r it

will

not

be

.

Rep

ort u

sed

for

Res

ourc

e A

lloca

tion

Yes

N

o, ra

ther

to in

form

de

cisi

on m

akin

g an

d to

en

sure

se

nior

m

anag

ers

have

a

cons

iste

nt p

ictu

re o

f ea

ch m

ultil

ater

al.

Yes

, to

rew

ard

good

pe

rform

ance

w

ith

addi

tiona

l fun

ds.

Not

mec

hani

cally

or

syst

emat

ical

ly,

but

can

influ

ence

pol

icy

dial

ogue

an

d fu

ndin

g de

cisi

ons.

Not

mec

hani

cally

or

syst

emat

ical

ly,

but

can

influ

ence

seni

or

man

agem

ent

deci

sion

-mak

ing.

No

Not

m

echa

nica

lly,

but

inte

nded

to

gu

ide

budg

et

allo

catio

ns

and

fund

ing

deci

sion

s.

Gen

eral

Str

engt

hs a

nd W

eakn

esse

s M

easu

rem

ent o

r M

easu

rem

ent

Mea

sure

men

t M

easu

rem

ent

Mix

ed

Mix

ed

Mix

ed

Mea

sure

men

t

45

AN

NEX

J: C

RO

SS-C

ASE

CO

MPA

RIS

ON

TA

BLE

OF

MU

LTIL

ATER

AL A

SSES

SMEN

T AP

PRO

ACH

ES

NA

ME

M

EFF

M

ES

� B

SC

ME

RA

Sc

orec

ard

PMF

RB

M

Ass

essm

ent

FAR

P

Man

agem

ent

mea

sure

men

t and

m

anag

emen

t m

easu

rem

ent a

nd

man

agem

ent

mea

sure

men

t and

m

anag

emen

t C

ompl

exity

of u

se

Fairl

y si

mpl

e to

co

mpl

ete

Che

cklis

t, ev

en

for

MO

s, ho

wev

er

data

an

alys

is /

rep

ortin

g be

com

e co

mpl

ex

and

time

cons

umin

g

Som

ewha

t com

plex

to

co

mpi

le

seco

ndar

y da

ta

give

n th

e va

riety

of

data

so

urce

s an

d tim

ing,

how

ever

less

tim

e co

nsum

ing

to

data

an

alys

is

for

indi

vidu

al M

Os.

Sim

ple

for

Des

k of

ficer

s to

an

swer

qu

estio

ns

with

no

ad

ditio

nal

prim

ary

da

ta

colle

ctio

n

requ

irem

ents

Scor

ecar

d som

ewha

t co

mpl

ex b

ecau

se o

f va

riety

of

da

ta

sour

ces.

MSS

eas

y fo

r Em

bass

ies

to

answ

er

but

time

cons

umin

g to

an

alys

e fo

r in

divi

dual

MO

s.

Com

plex

gi

ven

thre

e (3

) le

vels

, va

riety

of

da

ta

sour

ces

and

colle

ctio

n m

etho

ds.

Emba

ssy

surv

ey i

s fa

irly

sim

ple

but

time

cons

umin

g to

an

alys

e fo

r in

divi

dual

MO

s.

Com

plex

and

tim

e co

nsum

ing

to

cond

uct

requ

iring

be

twee

n 60

-100

p/d

le

vel

of

effo

rt fo

r tw

o pe

ople

ov

er

thre

e (3

) m

onth

pe

riod.

Fairl

y si

mpl

e sur

vey

appr

oach

, bu

t so

mew

hat

com

plex

gi

ven

it w

as th

e firs

t tim

e R

athe

r tim

e-co

nsum

ing,

Sust

aina

bilit

y N

ot fo

r sin

gle

dono

r N

ot fo

r sin

gle

dono

r Y

es

Yes

Y

es

Not

for s

ingl

e do

nor

Yes