Country-led Joint EvaluationCountry-led Joint Evaluation Dutch ORET/MILIEV ProgrammeDutch ORET/MILIEV Programme in China in China

Chen Zhaoying NCSTENCSTE , China

Hans Slot IOB, The Netherlands

Sixth Meeting of the DAC Network on Development Evaluation, 27-28 June 2007, Paris

2

Presentation OutlinePresentation Outline

■ A brief note on the evaluationA brief note on the evaluation

■ The response from stakeholders The response from stakeholders and tand the dissemination of the he dissemination of the evaluationevaluation

■ Issues for ssues for discussiondiscussion

3

Joint Evaluation of Aid Programs

DefinitionDAC Glossary defines joint evaluation as: An evaluation to which different donors and/or partners participate.

Four categories of joint evaluation

■ Donor + Donor

Donor + Partner Country

■ Multi-Donor + Multi-Partner

■ Partner + Partner

4

Joint Evaluation of Aid Programs

OverviewOverview

■Partnership in development leads to partnership in evaluation. Joint Evaluation has been on the international development agenda since the early 1990s.

■Country-led Joint Evaluation of ORET/MILIEV Programme in China is regarded by DAC Network study report as one of the important joint evaluation cases since 1990.

5

Country-led EvaluationCountry-led Evaluation

■ Ownership in development leads to ownership of evaluation.

■ The World Bank, UNDP, the DAC, as well as some donor governments, have been developing approaches to promote country-led evaluations (CLEs) and to increase partner country ownership.

■ The majority of evaluations of development aid programmes are still donor evaluations: Led by donors and are carried out to meet donors’ requirements

■ What is needed are evaluations from the perspective of the partner country. Country-led evaluations will provide that.

6

The Country-led JointThe Country-led JointEvaluation in ChinaEvaluation in China

■ About the title of the evaluation

■ The recipient country leads the process, supported by the donor and both sides are jointly responsible for the evaluation

The evaluation covers the period 1991-2003. A total project portfolio of 84 projects were approved. 40 projects were finished and 44 were ongoing. Of these 44, 17 completed the transaction but still had to produce the final report.

The grant amount of the transaction is about € 200 million, the total transaction amount of the evaluated portfolio € 470 million.

7

How is the Evaluation How is the Evaluation Initiated?Initiated?

■■ After the experiences of joint evaluation of aid After the experiences of joint evaluation of aid development, NCSTE would like to have a more development, NCSTE would like to have a more pronounced role in the development evaluation pronounced role in the development evaluation and IOB would like to be involved in a recipient-and IOB would like to be involved in a recipient-led joint evaluation.led joint evaluation.

■■ The joint evaluation was co-initiated by IOB and The joint evaluation was co-initiated by IOB and NCSTE.NCSTE.  

8

The Evaluation Aims toThe Evaluation Aims to

Assess to what extent the programme (through its Assess to what extent the programme (through its activities / projects) has fulfilled the policy objectives, activities / projects) has fulfilled the policy objectives, needs and priorities of the Netherlands and Chinaneeds and priorities of the Netherlands and China

Assess to what extent the programme (through its Assess to what extent the programme (through its activities / projects) has fulfilled the policy objectives, activities / projects) has fulfilled the policy objectives, needs and priorities of the Netherlands and Chinaneeds and priorities of the Netherlands and China

Verify whether the funds have been Verify whether the funds have been appropriately and efficiently usedappropriately and efficiently used Verify whether the funds have been Verify whether the funds have been appropriately and efficiently usedappropriately and efficiently used

Provide information for both the Provide information for both the Chinese and the Dutch that could be Chinese and the Dutch that could be used to improve the programme, as used to improve the programme, as well as for policy formulationwell as for policy formulation

Provide information for both the Provide information for both the Chinese and the Dutch that could be Chinese and the Dutch that could be used to improve the programme, as used to improve the programme, as well as for policy formulationwell as for policy formulation

9

Evaluation CriteriaEvaluation Criteria

■■ The relevance of the ProgrammeThe relevance of the Programme

■■ The efficiency of the ProgrammeThe efficiency of the Programme

■ ■ The effectiveness of the ProgrammeThe effectiveness of the Programme

■■ The impact of the projects implemented under The impact of the projects implemented under the Programmethe Programme..

10

How is the evaluation How is the evaluation organized?organized?

Governance StructureGovernance Structure

An equal governance structure was established An equal governance structure was established for this particular type of evaluation.for this particular type of evaluation. ■■ The Steering Committee (SC)The Steering Committee (SC)

Make decisions on the evaluationMake decisions on the evaluation

■■ Team Leaders (TL)Team Leaders (TL)

Organise the fieldwork and other tasksOrganise the fieldwork and other tasks

■■ The Reference Group (RG)The Reference Group (RG)

Provide advice and supportProvide advice and support

11

How is the evaluation How is the evaluation conducted?conducted?

FFour steps of the joint evaluationour steps of the joint evaluation

■■ Design/TORDesign/TOR

■■ Setting up & organizationSetting up & organization

■■ Implementation & Final Draft ReportImplementation & Final Draft Report

■■ Report dissemination/Report dissemination/Following up Following up

12

How Is the Evaluation How Is the Evaluation Conducted?Conducted?

Evaluation MEvaluation Methodologyethodology■ Desk Study: Cover all the 84 projects

■ Field Visits: 35 projects were chosen, 19 finished and 16 ongoing projects

■ Questionnaires: 68 were returned from end users and 60 from suppliers

■ Stakeholder Workshops: 17 provincial-level workshops were held during the field visits. More than 300 stakeholders were invited to participate in the workshops

■ Cross-verified & In-depth Analysis: The information in the fact sheets and the evidence collected from the field visits were cross-verified to support in-depth analysis; the database of initial evidence for the 35 visited projects has been built

13

The Responses The Responses from Stakeholdersfrom Stakeholders

■ The responses from Chinese stakeholders

■ The responses from Dutch stakeholders

■ The responses from other development agencies

■ The policy reaction in China and the Netherlands

14

The Dissemination The Dissemination of the Evaluationof the Evaluation

The dissemination has been given attention from the beginning. Some dissemination activities are planned and some are not predicted.

■The final evaluation reports published in both Chinese and English

■ The dissemination workshop in Beijing China, March 29, 2007

■ Communicate this evaluation case at the evaluation training courses (IPDET, ShIPDET and other national evaluation training workshops in China)

15

Comments for Comments for the Evaluationthe Evaluation

■ The key benefits of the evaluation

■ The limitation of the evaluation

■ Potential challenges of the country-led joint evaluation

■ Making the decision to undertake a joint evaluation

16

The key benefits ofThe key benefits of the the Country-led Joint Country-led Joint EvaluationEvaluation

■ Offer great opportunities for mutual capacity development

■ Increase ownership, the evaluation reflects better the voice of the partner countries

■ More relevant to the needs of both sides

■ Sharing the burden of work of evaluation

17

The limitation ofThe limitation of the the Country-led Joint Country-led Joint EvaluationEvaluation

Institutional limitation in China ■ A national evaluation system has not been set up

■ No one in-charge of the official policy reaction

■ Lack of special budget for evaluation

18

The Potential challengesThe Potential challenges of the of the Country-led Joint Country-led Joint EvaluationEvaluation

■ The biggest challenge comes from different evaluation cultures and evaluation systems

■ Ensure that the evaluation meets the needs of both countries

■ Build consensus to design and conduct evaluation (such as: Identifying the key Issues in writing TOR, choice of field study cases)

■ Engage the key stakeholders in the design, conduct, and interpretation of evaluation

19

The Potential challengesThe Potential challenges of the of the Country-led Joint Country-led Joint EvaluationEvaluation

■ Develop capacity of partner countries to do AND use evaluation

■ Ensure that evaluation is USED appropriately

■ Enable partner countries to evaluate themselves rather than having evaluation done to them.

20

Listen toListen to

your Opinions and Adviceyour Opinions and Advice ■ What do you think of this joint evaluation? What are the advantages and weaknesses?

■ How to promote the country-led joint evaluation as a successful evaluation model?

■ What is the driving force for this kind of evaluation?

■ What are the major barriers to What are the major barriers to the country-led joint evaluation ? Lack of demand or weak evaluation ? Lack of demand or weak evaluation capacity ? capacity ?

■ Donor’s role in CLEs, particularly in the “first generation” of CLEs, by nurturing the country’s demand and facilitating evaluation activities.

21

Thank youThank youAny Questions, Opinions & AdvicesAny Questions, Opinions & Advices… …… …

Sixth Meeting of the DAC Network on Development Evaluation27-28 June 2007, Paris