-
Evaluation section
UNICEF EAPROJune 2017
Regional Evaluation Strategy 2018-2021UNICEF East Asia and the Pacific
-
Copyright: United Nations Children’s Fund (UNICEF) East Asia and the Pacific Regional Office Evaluation SectionDate of Final Version: June 2017Cover photo: A young girl with a cooking pot over her head at the local market close to the Sin Tet Maw camp for internally displaced persons (IDPs) in Arakan State, Rakhine State, Myanmar, Saturday 8 April 2017. © UNICEF/UN061856/Brown
-
Evaluation section
UNICEF EAPROJune 2017
Regional Evaluation Strategy 2018-2021UNICEF East Asia and the Pacific
-
AcknowledgementsThis regional evaluation strategy and action plan is the result of hard work of the EAPRO Evaluation section, the COs in the East Asia and the Pacific, the Evaluation Office in New York as well as colleagues from the Regional Office for South Asia and the meaningful contribution from Michael Quinn Patton.
ii
-
ForewordDear colleagues,
In his opening statement at the June 2017 Executive Board meeting, Antony Lake, UNICEF’s Executive Director, indicated that “our evaluation function is helping design, target and deliver interventions that will make the biggest difference in children’s lives. Evaluations demonstrate what works and what does not, and help us build a strong evidence base to constantly improve our programmes”. By supporting organizational learning and accountability, evaluations help UNICEF to continually improve its performance and results. Good evaluations serve UNICEF’s mission and promote its mandate to protect and promote children’s rights.
Our Evaluation Policy defines an evaluation “as a shared function within UNICEF” and calls for regional offices, to develop regional strategies that move the role of evaluations beyond project accountability and contribute towards better programme results, organizational performance and institutional advocacy. Thus, I am pleased to share the East Asia and the Pacific “Regional Evaluation Strategy and action plan: 2018–2021” approved during our Regional Management Team (RMT) meeting in April 2017.
This Regional Evaluation Strategy has been designed to help UNICEF senior managers strengthen the evaluation function in the East Asia and Pacific region so that the organization generates good-quality evidence that informs policy, programming and advocacy and ultimately contribute towards better results for children.
Kind regards,
Karin Hulshof Regional DirectorEast Asia and the Pacific Regional Office
iii
-
Executive summary By supporting organizational learning and accountability, evaluations help UNICEF to continually improve its performance and results. Good evaluations serve UNICEF’s mission and promote its mandate to protect and promote children’s rights.
Our 2013 revised Evaluation Policy reflects UNICEF’s commitment to demonstrate results and improve performance, learning and accountability. The evaluation function is carried out at all levels of the organization and in all contexts, from humanitarian crisis to transition situations to more steady development environments.
The Evaluation Policy defines an evaluation “as a shared function within UNICEF” and calls for regional offices, under the leadership of the Regional Directors, to develop regional strategies that move the role of evaluations beyond project accountability and contribute towards better programme results, organizational performance and institutional advocacy.
In East Asia and the Pacific, the UNICEF Evaluation Office in New York, the Regional Office (EAPRO) and its country offices are to work together to strengthen the evaluation function. EAPRO, however, retains an oversight, guidance, technical assistance and quality assurance role so that evaluations managed or commissioned by UNICEF (regional office and country offices) uphold high-quality standards.
Purpose of the strategy
As noted in the Global Meta-Evaluation Report 2014, “Because UNICEF is decentralized in nature, its evaluations are generally commissioned and managed at the country office level. On one hand, such an arrangement helps ensure that report analyses remain highly focused on the national context, but on the other, this decentralized system makes it difficult to maintain uniform quality, high credibility and utility of the evaluations produced organization-wide.”
This Regional Evaluation Strategy was designed to help senior managers corporately prioritize the evaluation function so that the organization generates good-quality evidence that informs policy, programming and advocacy and ultimately contribute towards better results for children. It intends to contribute to improve country office evaluation planning, budgeting, implementation, dissemination and use of findings.
In April 2017, the Regional Management Team approved the Strategy and action plan, thus endorsing five priorities: (i) prioritize evaluations and embed the process into the results-based management cycle; (ii) introduce or strengthen quality assurance systems; (iii) reinforce UNICEF staff capacity development; (iv) support national evaluation capacity development; and (v) maintain independence and credibility of evaluation findings. This will trigger transformational learning and adaptive management within UNICEF.
iv
-
To achieve the strategic priorities, the UNICEF Regional Director and Representatives in the East Asia and Pacific region have agreed to:
· Allocate dedicated and qualified human and financial resources and set up effective management and governance structures that preserve the independence and impartiality of the evaluation function. EAPRO and country offices will allocate, on average, 1 per cent of their budgets to cover the evaluation function.
· Carry out a minimal number of evaluations per management plan cycle. EAPRO will conduct at least two evaluations during its new regional office management plan cycle (2018–2021). Larger country offices in the East Asia and Pacific region have agreed to conduct at least five evaluations per country programme cycle, while medium-sized and smaller country offices will carry out at least three evaluations.
· Systematically use evaluation findings for strategic decision-making, such as reorienting the country programme or adjusting the programmatic area objectives. When commissioning and conducting evaluations, EAPRO and country offices need to have a clear intention to use the resulting analysis, conclusions and recommendations to inform decisions and actions.
· Prioritize national evaluation capacity development initiatives that engage government and development partners. Within the 2030 Agenda for Sustainable Development, evaluations have been given elevated significance because of their utility in helping countries measure their progress towards achieving the targets of the Sustainable Development Goals.
Intended audience
The primary audience of this document is senior management in the East Asia and Pacific Regional Office and country offices. In addition to the regional director, the deputy regional director and the section chiefs, the country office representatives, deputy representatives and planning, management and evaluation staff as well as programme staff will find the Regional Evaluation Strategy of importance to their work. The Evaluation Office and Field Results Group, the Office of Research, the Office of Emergency Programmes and the programme division at headquarters comprise the secondary audience.
v
-
ContentsAcknowledgements ii
Foreword iii
Executive summary iv
Abbreviations vii
Context and the need for an improved evaluation culture 1
i. The changing developmental paradigm gives a central role to evaluations 1
ii. Overview of the UNICEF evaluation function in the East Asia and Pacific region 2
iii. What do country offices request in terms of regional office support and technical assistance to improve the evaluation function? 6
Regional Evaluation Strategy 9
iv. What does the region need to prioritize? 9
v. How? The way forward. 11
Action plan 16
Process 16
Impact statement 16
Outcome statement 16
Intermediary outcomes 16
Specific outputs 16
Annexes 26
Annex 1. UNICEF accountabilities to evaluate at the regional and country levels 27
Annex 2. Comments on UNICEF country offices progress and challenges in the East Asia and Pacific region, 2015–2016 29
Annex 3. GEROS-reviewed completed evaluations 32
Annex 4. UNICEF [country office]: Standard operating procedures for better evaluations (Draft – 19 June 2015) 34
Annex 5. Analytics of the requests received in 2016 42
List of figures
Figure 1: Theory of Change on how to strengthen the UNICEF evaluation function in the East Asia and the Pacific region 17
vi
-
AbbreviationsAPEA Asia Pacific Evaluation Association
CEP costed evaluation plan
CO country office
CP country programme
CPD Country Programme Document
DREAM Data Research Evaluation and Monitoring Annual Meeting
DROPS deputy representatives and operations
EAPRO East Asia and the Pacific Regional Office
EMOPS Office of Emergency Programmes
EO Evaluation Office
GEROS Global Evaluation Reports Oversight System
IMEP Integrated Monitoring and Evaluation Plan
JPO Junior Professional Officer
M&E monitoring and evaluation
MR management response
NECD National Evaluation Capacity Development
OECD Organisation for Economic Co-operation and Development
PME planning, monitoring and evaluation
PRIME Integrated Monitoring Evaluation and Research Planning
QA quality assurance
RBM results-based management
RD regional director
RMT Regional Management Team Meeting
RO regional office
ROMP Regional Office Management Plan
ROSA Regional Office for South Asia
SOP standard operating procedures
UNDAF United Nations Partnership Development Framework
UNDP United Nations Development Programme
UNEDAP United Nations Evaluation Development for Asia and the Pacific
UNEG United Nations Evaluation Group
UNICEF United Nations Children’s Fund
UNITAR United Nations Institute for Training and Research
WASH water, sanitation and hygiene
vii
-
8-day-old son (no name yet) at Marara Clinic in Honiara. Sabina came for general check ups of her baby and holds him while she waits, Solomon Islands/2017© UNICEF/UN062221/Sokhin
viii
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–2021 1
Context and the need for an improved evaluation culture
i. The changing developmental paradigm gives a central role to evaluations
1. Despite the various breakthroughs that the Millennium Development Goals achieved, it became evident late in that experience that the shortfalls were partly due to the absence of appropriate monitoring and evaluation systems. The next iteration of development targets would not be remiss. During the 2015 United Nations Evaluation Group’s (UNEG) Norms and Standards for Evaluation High-Level Group Event, the former United Nations Secretary-General recognized that “evaluation is everywhere and, at every level, will play a key role in implementing the new development agenda”. Thus, as the 17 goals came together within the 2030 Agenda for Sustainable Development, the evaluation function became an imperative for performance measurement, learning and general accountability of the development paradigm.1
2. United Nations Member States also recognize that evaluations are a core function in their development processes because they help strengthen and support development results.2 And development partners accept that they need to generate and use evidence to demonstrate that they are achieving results.
3. This shift towards greater learning and accountability represents opportunity for UNICEF to advocate for independent, credible, good-quality and useful evaluations for evidence-based policy-making at the global, regional, national and local levels. Evaluation findings should inform the implementation, follow up and review of progress towards the SDGs at the global and national levels. National development policies need to be informed by credible and independent evidence. To do so properly, adequate national government, bilateral and multilateral donors’ resources need to be invested.
1 According to the General Assembly draft outcome document on the post-2015 development agenda.2 United Nations General Assembly resolution A/RES/69/237 on capacity building for the evaluation of development activities at the country level.
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–20212
ii. Overview of the UNICEF evaluation function in the East Asia and Pacific region
The 2013 UNICEF revised Evaluation Policy governs the organization’s evaluation function and provides a comprehensive framework for all evaluation activities we undertake. The policy states that evaluations “unequivocally serve the organization’s mission and supports UNICEF in fulfilling its mandate”. By supporting organizational learning and accountability, evaluations help UNICEF continually improve its performance and results. As the policy notes, evaluations in UNICEF serve “to support planning and decision-making and to provide a basis for informed advocacy—aimed at promoting the well-being of all children, everywhere.” In focusing on the substantive rationale, value and performance of interventions and institutional functions, evaluations improve results and stakeholder satisfaction. This function is carried out at all levels of the organization and in all contexts, from humanitarian crisis to transition situations to more steady development environments.
The policy also acknowledges that evaluations at the regional and country levels are especially important because they provide reliable evidence to inform decision-making within UNICEF and among its partners and stakeholders and for well-founded advocacy and advice. The Evaluation Policy calls for regional offices, under the leadership of the respective regional directors, to develop regional strategies and engage senior management attention in the Regional Management Team (RMT) and elsewhere. The policy regards the evaluation practice “as a shared function within UNICEF”.3 Roles are distributed across senior leaders and oversight bodies, heads of offices, technical evaluation staff and sector-based programme staff. Accountabilities are distributed at (i) the headquarter level, (ii) regionally and (iii) the country level.4
34
4. The UNICEF Evaluation Office in New York, its East Asia and Pacific Regional Office (EAPRO) and its country offices generally collaborate to strengthen the organization’s evaluation function. The regional office has an oversight, guidance, technical assistance and quality assurance role, aiming to ensure that the evaluations managed or commissioned by UNICEF (regional office and country offices) uphold the high-quality standards set for them. The regional office and country office evaluation activities also include developing nationally and regionally specific evaluation strategies, engaging in partnerships for evaluation and supporting national evaluation capacity development.
5. Because it is an institutional priority, the evaluation function has been established over time in all country offices. With EAPRO 2014–2017 priorities aimed at strengthening the use of the evaluation function “to support evidence-based and critical decision-making at the programmatic and policy level”, the quality of evaluations being conducted (Annexes 2 and 3) and the use of findings has been steadily improving.5
3 UNICEF (2011) defines an evaluation as a “judgement [on] the relevance, appropriateness, effectiveness, efficiency, impact and sustainability of development efforts, based on agreed criteria and benchmarks among key partners and stakeholders. It involves a rigorous, systematic and objective process in the design, analysis and interpretation of information to answer specific questions. It provides assessments of what works and why, highlights intended and unintended results, and provides strategic lessons to guide decision-makers and inform stakeholders.”
4 For details, see the revised Evaluation Policy of E/ICEF/2013/14, pp. 7–10.5 For example, the 2015 Malaysia equity evaluation, the Timor-Leste water, sanitation and hygiene evaluation and the Viet Nam mother tongue
evaluation.
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–2021 3
6. But the improvements are uneven across the region, with the foundations of the evaluation function and the quality, credibility and use of evaluations findings still weak in several country offices. The following discusses the continuing challenges to a strong evaluation function in UNICEF’s work as well as opportunities to reach the level of quality required.
Challenges
7. There is a need for a plan to strengthen the evaluation function generally.6 There is a proliferation of strategies across UNICEF,7 and the level of effort needed to roll them out within the organization is challenging because they all demand dedicated resources, proper systems and processes.
8. With few exceptions, evaluations tend not be used as the basis for strategic decision-making (such as reorienting the position of the country office or the country programme). Several country offices still plan their evaluations on an annual basis, drafting their Integrated Monitoring and Evaluation Plan essentially as a wish list. Project-level evaluations prevail, generally driven by bilateral donors’ demands for upwards accountability. This often triggers “evaluation fatigue”. To overcome this, better planning and prioritization and better use of evaluation findings are critical.8
9. Despite country office efforts, dedicated and qualified professional human resources for planning and managing evaluations and overseeing the quality and use of deliverables are limited. Country office planning, management and evaluation (PME) staff9 and monitoring and evaluation staff continue to dedicate most of their time to planning and monitoring and are left without proper time and resources to plan and manage evaluations or to properly promote use of the findings. This, coupled with the downsizing of many country offices, is affecting evaluation capacity, with monitoring and evaluation posts being cut or downgraded. Country offices tend to overcome this human resource deficit by engaging sector programme staff in the management of evaluations. But these individuals tend to be unfamiliar with the UNEG-defined norms and standards for evaluations, which can jeopardize the evaluation function’s credibility. This is also affecting the independence and impartiality of the evaluation standards set in the Evaluation Policy, with programme managers evaluating their own programmes. A recent self-assessment found that only 22 per cent of country offices globally have an environment in which PME or monitoring and evaluation staff report to the country representative. Many staff report to the planning and monitoring staff in charge or the deputy country representative, and 23 per cent report to a section chief, with roles and responsibilities interpreted differently across country offices, despite the guidance provided by the Evaluation Office.
6 Global Evaluation Committee, June 2015.7 As noted during the September 2014 Global Evaluation Committee meeting.8 According to the Evaluation Policy, a country office needs to ensure an evaluation is undertaken: (a) before a programme replication or scaling up
(pilot initiatives); (b) when responding to major humanitarian emergencies; (c) following long periods of unevaluated programme implementation, especially when the programme has been implemented for at least five years without any evaluation activity; (d) when expenditure for each outcome has reached US$10 million; and (e) when the average annual expenditure for each outcome exceeds US$1 million.
9 According to a 2011 global survey, PME staff only dedicate 14 per cent of their time to evaluations. This limited time for evaluations was noted during the June 2015 deputy representatives and operations meeting.
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–20214
10. An alternative approach recently tested in three country offices (Cambodia, Malaysia and Myanmar) is a specialist evaluation staff position reporting directly to the representative to ensure independence from the programmes and making technical reports to the regional evaluation adviser.10 As pointed out in a recent exploratory study on the decentralized evaluation functions across UNEG agencies, this approach can boost evaluation capacity at the country level and promote efficiencies. The multi-country approach allows the sharing of costs between country offices. Although more coordination is required, the approach allows staff to positively influence the evaluation system and culture of country offices. It also allows greater consistency, access to resources and the sharing of monitoring and evaluation tools. And it facilitates replication of good practices. By technically reporting to the regional evaluation adviser, the specialist is in a better position to implement the regional strategy at the country level. Before engaging further in shared posts, however, the human resources section is evaluating whether this option could be more systematically applied in our region and in others.11
11. Evaluation teams are often led by consultants with sound technical sector expertise but with limited evaluation experience. Teams that are not familiar with good evaluation methods and UNEG’s quality standards can produce poor-quality reports, especially when evidence is not sufficiently triangulated. Several evaluation reports submitted to the regional office for quality assurance, for example, read more like progress reports than a proper independent and evidence-based evaluation. This improper format inhibits adequate learning and accountability at both the regional and national levels.
12. There is still need for quality assurance and effective use of the evaluation findings. Often the purpose and objectives of the evaluation are not always shared at the country office level (as reflected in the terms of reference); stakeholders are not involved throughout the evaluation process, thus limiting the level of ownership and active engagement. As noted in a recent meta-evaluation, “Because UNICEF is decentralized in nature, its evaluations are generally commissioned and managed at the country office level. On one hand, such an arrangement helps ensure that report analyses remain highly focused on the national context, but on the other, this decentralized system makes it difficult to maintain uniform quality, high credibility and utility of the evaluations produced organization-wide.”12 Emerging good practices in UNICEF’s work especially need to be more robustly documented through evaluations.
13. There are no indicators to determine the use of evaluation findings for advocacy purposes or as inputs for programming and other decision-making, even though the evaluation management response submission rate has reached 100 per cent, and the completion rate of actions required has steadily increased.
10 This approach allows country offices to have evaluation specialists report to the representative while programme managers report to the deputy representative. This appears to be a successful option when roles and responsibilities of the shared evaluation post are articulated by each country office in relation to other PME or M&E staff. Other country offices, such as Lao PDR, Thailand and Viet Nam, initially considered establishing a similar shared post but soon realized that they did not have sufficient resources to fund the position for at least two years.
11 Other options that could be considered would be that the Social Policy section takes the lead on the PME function, supplemented by a national officer, technical assistance and ad hoc consultancies for managing and providing quality assurance of evaluations.
12 GEROS: Global Meta-Evaluation Report 2014, Universalia (2015), p. 2.
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–2021 5
Opportunities
14. Coverage and quality of evaluations is progressively improving in the region. In the past three years, 13 of the 14 country offices completed at least one evaluation.13 According to the 2017 evaluation office report to the UNICEF Executive Board, the quality of country office evaluations in the East Asia and Pacific region have progressively improved.
15. The average budget use for evaluation in the region has skyrocketed, going from 0.2 per cent in 2014 to 1.8 per cent in 2016.14 Over this period, East Asia and the Pacific progressed from the second-lowest ranking region in terms of budget use for evaluations to the highest rank. The number of country offices spending more than 1 per cent of their programme expenditure quintupled between 2014 and 2016. Despite that staggering progress, unevenness prevails in the region; some country offices spend 3 per cent of their budget for evaluations, while the regional office only dedicates 0.1 per cent.
16. Since 2014, a costed evaluation plan accompanies every Country Programme Document (CPD),15 thus anchoring the evaluation function in UNICEF’s results-based management cycle. In 2017, a total of 11 country office CPDs will have a costed evaluation plan (such as Cambodia, Indonesia, Thailand and the Philippines). This should allow the country offices to take a more strategic medium-term approach for ensuring programmatic coverage and progressively engage UNICEF to support country-led evaluations.
17. See Annex 2 for more detailed comments on UNICEF country offices progress and challenges in the East Asia and Pacific region, 2015-2016.
13 EAPRO has not completed a regional evaluation since 2013, although it did co-manage and quality monitor two bi-regional evaluations with ROSA in 2016.
14 Only three other regions spend more than 1 per cent: Latin America and Caribbean Regional Office, at 1.4 per cent; Eastern and Southern Africa Office, at 1.3 per cent; and the Regional Office for South Asia, at 1.1 per cent.
15 Costed evaluation plans will be developed for every new UNDAF, which is an important development for those countries in the region that have a common programme of cooperation with their host government.
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–20216
iii. What do country offices request in terms of regional office support and technical assistance to improve the evaluation function?
18. Most country offices’ requests seek guidance on planned and ongoing evaluations and for quality assurance of evaluation deliverables. In 2016, the Evaluation section provided support, quality assurance and comments to more than 94 evaluation deliverables (see Annex 5), including terms of references and inception, draft and final evaluation reports from country offices in the East Asia and Pacific region, bi-regional and global evaluations. An assessment of those items indicate that quality assurance mechanisms are not in place at the country office level. With few exceptions, country offices have neither established a peer review group nor a management group to provide proper quality assurance.16 To address this systemic issue, the Evaluation section provided guidance for the development of the UNICEF Cambodia Standard Operating Procedures for Better Evaluation (see Annex 4). After being piloted in the Cambodia Country Office, those standard operating procedures (SOPs) were used and adapted by other country offices in the region (such as the Democratic People’s Republic of Korea, Malaysia, Mongolia and Myanmar). These SOPs can now be adapted to help country office management and staff ensure that evaluations are well planned and managed on time and on budget and that they produce credible, relevant and useful reports.
19. Country offices have often asked for help in professionalizing UNICEF and other UN staff through capacity development. Because staff competencies tend to vary and staff turnover is high,17 developing and facilitating specific training for UNICEF staff and other UN staff on the evaluation function’s core components has been the second-most frequent request. In response, capacity development sessions have been organized to develop UNICEF staff and partner staff capacities in the Democratic People’s Republic of Korea, the Lao People’s Democratic Republic, Mongolia, Myanmar, Papua New Guinee, Philippines and Viet Nam. Additionally, sessions on the new UNEG norms and standards, evaluability and evaluation management were facilitated at the joint UN Evaluation Development Group for Asia and the Pacific (UNEDAP) in the 2015 and 2016 training on “Evaluation in the UN Context”. Together with the Regional Office for South Asia and the Evaluation Office in New York, EAPRO organized joint evaluation network meetings in Kathmandu and Bangkok. These events contributed towards increasing staff capacity to manage and use evaluations as well as to ensure coherence with the evaluation function at the global level and with other UN agencies.18 In the future, country office PME staff and dedicated evaluation staff could support each other through peer reviews and training that would further contribute towards developing professional competencies.
16 Quality assurance for two final evaluation reports (on the Thailand Country Office’s National Child and Youth Development Plan and the Lao PDR WASH country programme) was requested three times for each. This shows that, even when a review team was set up, standardized procedures, quality assurance processes and mechanisms were not effectively working at the country office level. The regional evaluation adviser recommended these two country offices look to what extent the consultants’ team had addressed comments previously shared before sending the deliverables to the regional office. In the case of the Thailand Country Office evaluation, the regional evaluation adviser met the team leader and participated in the debriefing to provide direct advice.
17 Ian C. Davies and Julia Brummer: Final Report to the UNEG Working Group on Professionalization of Evaluation, Geneva, 2015; and UNEG: Evaluation Competency Framework, Geneva, 2016.
18 Beyond the previously noted trainings, in-house capacity development on evaluations is de-prioritized, with no country office learning plan prioritizing this critical function. This is partly because the previous regional evaluation adviser thoroughly supported country office capacity development needs.
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–2021 7
20. Technical assistance represents the third most frequent request from country offices. As opposed to other country office requests, this is the most diverse in nature. Support has ranged from hands-on guidance on an after-action review of UNICEF’s emergency response to cyclone Pam to guidance on a country programme evaluation (Indonesia and Philippines) as well as on United Nations Partnership Development Framework (UNDAF) evaluations for Cambodia Fiji, Papua New Guinea and Viet Nam.
21. Technical assistance has been given on how to develop and prioritize evaluations in the costed evaluation plans in the Democratic People’s Republic of Korea, the Lao People’s Democratic Republic, Mongolia, Thailand, Viet Nam and Pacific island country offices. The Cambodia, Indonesia, Malaysia, Mongolia, Philippines and Viet Nam country offices have requested regional office support for their ‘evaluability’ assessments,19 although guidance is still under development. EAPRO has assisted in a review of the adequacy of the Indonesia country programme design and the availability of data and systems to carry out an evaluation as well as to understand whether stakeholders are on board to do an evaluation and whether they have sufficient resources available to do an evaluation. The regional office evaluability assessment support may trigger country office senior management buy-in for conducting more strategic evaluations at the outcomes level.
22. National evaluation and partner capacity development represents the fourth-most frequent request. Despite the strong emphasis that UNICEF places on developing national evaluation capacity, which includes not only strengthening the evaluation systems of national governments but also those of civil society partners, this demand is nascent. To accommodate the growing requests, partnerships with other UN agencies have been critical (mainly the United Nations Development Programme (UNDP) and UN Women) and development partners (Asian Development Bank, the World Bank and the Asia Pacific Evaluation Association) because it’s an area that is broader than UNICEF’s core mandate and priorities.
19 The OECD’s Development Assistance Committee defines an ‘evaluability’ assessment as “the extent to which an activity or a project can be evaluated in a credible fashion. Based on country office demand, the REA supports evaluability studies. These may enable UNICEF to save resources and correct the design flaws and to understand whether data and the environment is conducive before launching an evaluation.
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–20218
Boys play on a frozen body of water, in the ‘soum’ (district) of Ulaan-Uul in the northern Khövsgöl ‘Aimag’ (province), Mongolia/2012© UNICEF/UNI134453/Sokol
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–2021 9
Regional Evaluation Strategy23. The Regional Evaluation Strategy focuses on what the regional office and country offices can
do (their respective roles) to reinforce the evaluation function, especially the use of evaluation findings. The strategy provides guidance for conducting high-quality evaluations that inform senior management decision-making and respond to country office and regional office learning and accountability needs. The strategy aims to foster the credibility, use and quality of evaluations in a highly decentralized organization.
24. The strategy’s 2021 goal is to have an evaluation function that generates useful evidence that strategically informs policy, programming and advocacy and thus contributes towards better results for children. Of critical importance to the strategy is the involvement of children and young people throughout the evaluation activities across the region. Ultimately, the strategy envisions that evaluations will trigger transformational learning and adaptive management within the organization and among its partners.
iv. What does the region need to prioritize?20
25. To improve the evaluation function across the region, the strategy targets five strategic priorities: (a) prioritizing evaluation and embedding the function in the results-based management cycle;21 (b) strengthening the quality assurance system; (c) reinforcing the regional office and country office internal evaluation capacities; (d) supporting national evaluation capacity development; and (e) maintaining independence and fostering credibility and use of findings. Following through on these five priorities will nurture a stronger evaluation culture throughout UNICEF and among its core partners.
26. Prioritizing evaluation and embedding it in the results-based management cycle: When evaluations are better understood as a core component of results-based management, they will be better planned and of better quality and utility to UNICEF. When commissioning evaluations and conducting an evaluation, the regional office and country offices should have “a clear intention to use the resulting analysis, conclusions and recommendations to inform decisions and actions”.22
20 Regional offices, under the leadership of the regional director, provide regional leadership in (a) governance and accountability (especially in developing regional strategies and engaging senior management), (b) guidance and quality assurance, (c) conducting evaluations, (d) partnerships for evaluation, (e) development and professionalization of the UNICEF evaluation function and (f) national evaluation capacity development. For more details, see the UNICEF Evaluation Policy, p. 9.
21 By linking it more strongly to strategic positioning and planning.22 See norm 2 in UNEG: Norms and Standards for Evaluation, Geneva, 2016, .
http://www.unevaluation.org/document/detail/1914
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–202110
27. The regional office and country offices must take a strategic approach to evaluations to ensure adequate coverage and a medium-term perspective in their respective costed evaluation plan. The Evaluation section can provide guidance and support towards improving country office evaluation planning, budgeting, implementation, dissemination and use of findings. It can also review the planned evaluations and evaluation priorities with representatives, deputy representatives and PME sections. It can help country offices articulate their evaluation scope and purpose (organizational learning and improvement, accountability, transparency and increased use for evidence-based advocacy and decision-making).
28. Strengthening quality assurance system: The EAPRO Evaluation section can assist country offices in designing, managing and monitoring the quality of evaluations against the UNEG norms and standards. Systems, such as SOPs (Annex 4), can be adapted by country offices and applied throughout all phases of their evaluations.
29. When needed, the EAPRO Evaluation section can also help clarify roles and responsibilities of the country offices, the regional office and headquarters: who is accountable for the evaluation function and who manages them. Country offices need to identify adequate financial and human resources and procedures to ensure that evaluation quality and use of findings, conclusions and recommendations meet the minimum standards.
30. Reinforcing UNICEF staff capacity: When the capacity of PME and programme staff to manage and quality assure evaluations is weak, the EAPRO Evaluation section can support the recruitment of evaluation specialists or managers and support the regional office and country office capacity development initiatives. Training and coaching of staff are provided as per country office requests.
31. Internally: Qualified national and international resources are to be recruited to dedicate appropriate time to implement the evaluation function. When budget constraints are present or the volume of planned individual country evaluations is likely to increase, a shared evaluation specialist post could be an option that neighbouring countries consider. The regional Evaluation section can support country offices’ (i) recruitment processes by participating in interview panels, (ii) coaching staff and extending other capacity-building activities. Country offices can use the Human Resource Development Plan, including capacity building for national staff.
32. Externally: To carry out evaluations, the regional office and country offices contract qualified independent evaluators, supported by sector experts when needed. The Evaluation section can provide an up-to-date quality-controlled roster of external evaluators and firms that carry out high-quality evaluations. The market for the international development evaluation suppliers in the region is recognized as underdeveloped. When engaging local or regional suppliers, UNICEF country office staff should make sure they are aware of the UNEG evaluation standards and expectations on all evaluation products.
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–2021 11
33. Supporting national evaluation capacity development: In the post-2015 development priorities, UNICEF will further contribute toward improving national capacity to conduct country-led evaluations. Country offices, with the support of the regional office, can identify supply and demand as well as partners and priority actions for national evaluation capacity development. The strengthening of national capacities should involve working with other UN agencies, bilateral donors, government ministries (such as planning and finance) and universities. Country office road maps will need to be established.
34. Maintaining independence credibility and use of evaluations: Independence of evaluation activities is necessary for their credibility, which in turn underpins the use of evaluation findings, conclusions and recommendations. It allows evaluators to be impartial and free from undue pressure. As outlined in norm 4 of the UNEG, “The independence of the evaluation function comprises two key aspects—behavioural independence and organizational independence.” Considering the highly decentralized nature of the evaluation function, it is critical to preserve this degree of independence by separating the roles and responsibilities for the evaluation function within the country office. Those responsible for the evaluation function should report directly to the country representative. This arrangement mirrors the regional office set-up, with the evaluation advisor reporting to the regional director.
35. Considering that conducting evaluations represents a growing investment in the region, intentionality and use of findings, conclusions and recommendations are critical to consider throughout the evaluation cycle. Active involvement of stakeholders helps to boost their ownership and trigger learning with the organization and among external stakeholders, including children, youth, civil society, government and donors.
v. How? The way forward.
36. To strengthen the evaluation function at the regional office and in country offices, political leadership and adequate funding are needed, together with clear norms, mechanisms and expectations.
37. All stakeholders in the regional and country offices need to corporately prioritize evaluations. Country representatives and deputy representatives have relevant evaluation targets in their own plans and performance reviews. Large country offices should conduct five evaluations per country programme cycle, while medium-sized and smaller country offices should carry out at least three evaluations over the same period.23 The regional office has committed to at least two evaluations in the new Regional Office Management Plan.
23 In the East Asia and Pacific region, large country offices have more than $12 million in operational resources per year, while medium-sized and small country offices have less than 12 million OR. Country offices with more than $20 million in operational resources should allocate 3 per cent of their budget to the evaluation function.
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–202112
38. The Evaluation section can help EAPRO and country offices to plan and budget their evaluations when the CPD and costed evaluation plan are being developed. The region’s combined costed evaluation plans should allocate an average of 1 per cent of programme expenditure to the evaluation process. By allocating adequate human and financial resources and setting up effective management and governance structures, the independence and the impartiality of the evaluation function can be preserved. Each evaluation report will be supplemented with a management response that will be implemented.
39. Evidence from recent evaluations should be systematically incorporated into the new CPD. Knowledge management initiatives, such as the Strategic Moments of Reflection, the Annual Synthesis, the Evaluate newsletter, the UN Evaluation day and joint network meetings, are to be taken forward to support the dissemination and adoption of evaluation lessons. Joint regional network meetings are to be arranged every 18 months.
40. The regional and country offices need to develop a wider learning agenda and establish peer learning groups. To help fulfil existing knowledge gaps on emerging evaluative practices (evaluability assessments, developmental evaluations and national evaluation capacity development), good practices on evaluations that make a difference for children in the region must be regularly documented.
Integrating evaluation with results-based management
41. Evaluation findings, conclusions and recommendations are to be used for organizational learning, informed decision-making and accountability. A mechanism should be established to ensure that strategic decisions (such as reorienting the position of the country office or the country programme) at the regional and country office levels require evidence from evaluations of past interventions
42. Evaluation needs are to be more explicitly embedded in results-based management, with emphasis on CPD evaluability assessments, through theories of change, well-defined results, SMART indicators and the consistent establishment of baselines and monitoring systems. Evaluability assessments could improve UNICEF’s understanding of the adequacy of the programme design perspective, the availability of data and information to carry out an evaluation and guidance on possible approaches to evaluations. Once headquarters finalize the guidance, the regional office will share it and a checklist on evaluability with country offices.
43. Country offices should allocate adequate time and resources to planning and managing evaluations. Staff with relevant skill sets must manage and provide adequate guidance to consultants. Considering the representatives’ accountability for the evaluation function at the country level, they should allocate commensurate resources that are in line with the host government’s evaluation capacity and the size of the country programme. Country offices should include the evaluation function in the job description of their PME staff to report to the country representative.
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–2021 13
44. All EAPRO and country office evaluations are to be adequately managed and quality checked against the UNEG norms and standards. The quality assurance mechanism is to be strengthened to ensure that there is improved planning, implementation, use, dissemination and monitoring of the uptake of evaluation results, both at the regional office and country office level. Country offices should adapt and incorporate the SOPs for better evaluations (Annex 4) that were trialled in South-East Asia. Considering the number of evaluations, the EAPRO Evaluation section will prioritize the most strategic evaluations, based on relevance and budget and ask country offices to start progressively setting aside a proportion of their funding to use existing CEECIS, MENA and ROSA long-term agreement24.
45. Additionally, indicators to determine the use of evaluation findings for advocacy purposes are to be defined (explicit inputs for into programming and decision-making) and captured. Country offices and national partners should follow ethics review standards and procedures when conducting research, studies and evaluations. Once the research strategy is completed, an ethical board should be established to review evaluations and research results.
46. Within the East Asia and Pacific region, UNICEF will give attention to its internal capacity development as well as the capacity development needs of UN agencies and other partners. UNICEF capacity development is based on country office demands and met through available online training resources as well as specific in country activities. UN agencies capacity development will continue through joint UN training in Asia and the Pacific with additional focus on UNDAF evaluations. The government capacity development is described further on. UN system-wide support is required for national evaluation capacity development.
47. UNICEF will support rigorous and evidence-based country-led evaluations by helping to strengthen national data systems and national evaluation capacity. UNICEF (together with other UN agencies), in accordance with General Assembly resolution A/RES/69/237 on capacity building for the evaluation of development activities at the country level, will support “upon request efforts to further strengthen the capacity of Member States for evaluation, in accordance with their national policies and priorities”. UN agencies should work towards a common national evaluation capacity objective and should apply a systemic and synergistic approach to assisting countries. In each country, UNICEF, together with the UN system, should identify each agency’s comparative advantage; we can then leverage that advantage to maximize results. By partnering with other UN agencies and development actors, national evaluation capacities will be strengthened through the mapping of existing development partners’ supply and national government demand to reinforce their evaluation function.
24 Other regional offices outsource quality assurance to private companies and universities through global and regional long-term agreements. In EAPRO, financial resources are not currently available for this function; rather, it is being implemented by country offices and the regional evaluation adviser. Indicatively, country offices should set aside 1–5 per cent of budget resources for evaluations and knowledge generation, including quality assurance.
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–202114
48. Country offices should participate in diagnostic studies and stakeholder mapping to identify actors and entry points. EAPRO, the country offices and UNICEF headquarters can support member States and partners to mainstream evaluation through:
· awareness raising and advocacy;
· knowledge sharing of existing good practices and policies (in Malaysia, the Philippines and Thailand, where national evaluation policies have been developed with UNICEF support);
· capacity development; and
· evaluation action plan development.
49. The 2030 Agenda for Sustainable Development and aid effectiveness reform promote national ownership, alignment as well as evidence-based decision-making. National evaluation strategies could be developed as per country office demands. To keep track, UNICEF as well as others should report on its implementation.
50. During the first quarter of 2017, UNICEF EAPRO and UNDP Asia-Pacific regional Office decided to identify emerging national evaluation capacity development practices in the region by jointly launching a series of country case studies. The initial phase will include five country case studies—Indonesia, Malaysia, Thailand, Sri Lanka and either Nepal or Philippines—with further country case studies to be initiated over the course of 2018. A regional synthesis report on emerging good practices will be developed, based on the country case studies. This will serve to showcase existing national evaluation champions and emerging country practices in the region, distil key success factors, trends and lessons learned. Participating in this study will help UNICEF country offices understand what national evaluation capacity there is in terms of infrastructure and with which strategic partners UNICEF could further work. This may also foster South–South cooperation.
51. Monitoring and review. Progress on the Regional Evaluation Strategy and action plan will be reviewed every two years by the RMT. The action plan will be monitored on an annual basis by the regional evaluation section, and progress reports will be provided to the Regional Director and the RMT.
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–2021 15
A girl washes her hands at a UNICEF-provided water point at a new elementary school, built with UNICEF assistance, in the village of Neusok Teubaluy in the district of Aceh Besar, Indonesia/2007© UNICEF/UNI48741/Estey
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–202116
Action plan
Process:The implementation plan was drafted in late 2016 and incorporated two rounds of country office comments. The plan was then presented and validated at the Joint EAPRO ROSA Evaluation Network meeting in March 2017 and then endorsed at the RMT in April 2017. The Theory of Change below was subsequently developed.
Impact statementEvaluations make a difference in children’s lives
Outcome statement The UNICEF East Asia and Pacific Regional Office and country office evaluation function is corporately prioritized and strengthened.
Intermediary outcomesThe evaluation function contributes to UNICEF’s organizational learning, informed decision-making and accountability for results.
Quality, credibility and utility of the evaluations are improved through better planning, implementation, quality assurance, dissemination and use of evaluations as well as to staff and partners’ capacity development.
To achieve these outcomes, a series of outputs and a set of actions that, respectively, the regional office and country offices should prioritize. Because these are process components, some specific indicators are proposed. Yet, overall progress and performance will be measured against global key performance indicators, as reported in the global dashboard.
Specific outputs These are directly linked with the strategy priorities.
· Evaluation function is systematically embedded in UNICEF’s results-based management. · Evaluations are planned with an annual and multiple-year horizons. · The evaluation function at the regional office and the country offices is adequately resourced.· All EAPRO and country office evaluations are adequately managed and quality assured
against the UNEG Norms and Standards.· EAPRO and country offices actively foster evaluation use.· EAPRO and country offices prioritize national evaluation capacity development.· EAPRO and country offices strategically position UNICEF with regional United Nations
interagency evaluations.· EAPRO regularly interacts with the Evaluation Office.
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–2021 17
Fig
ure
1: T
heo
ry o
f C
han
ge
on
ho
w t
o s
tren
gth
en t
he
UN
ICE
F ev
alu
atio
n f
un
ctio
n in
th
e E
ast
Asi
a an
d t
he
Pac
ific
reg
ion
Sp
ecifi
c o
utp
uts
Inte
rmed
iary
o
utc
om
es
Ou
tco
me
stat
emen
t
Imp
act
stat
emen
t
Sys
tem
atic
ally
em
bed
ded
in
UN
ICE
F’s
resu
lts-
bas
ed
man
agem
ent
Ad
equ
atel
y re
sou
rced
An
nu
al a
nd
m
ult
iple
-yea
r h
ori
zon
s
Ch
ildre
n, d
uty
b
eare
rs &
ri
gh
ts h
old
ers
invo
lved
Act
ivel
y fo
ster
use
of
eval
uat
ion
fi
nd
ing
s
Pri
ori
tize
n
atio
nal
ev
alu
atio
n
cap
acit
y d
evel
op
men
t
Reg
ula
rly
inte
ract
wit
h
the
Eva
luat
ion
O
ffice
UN
DA
F ev
alu
atio
n
qu
alit
y as
sura
nce
Ad
equ
atel
y m
anag
ed
& q
ual
ity
assu
red
Eva
luat
ion
s m
ake
a d
iffe
ren
ce in
ch
idre
n’s
live
s
By
2021
, UN
ICE
F ev
alu
atio
n f
un
ctio
n is
co
rpo
rate
ly p
rio
riti
zed
and
str
eng
then
ed a
t th
e E
AP
RO
an
d c
ou
ntr
y o
ffice
leve
ls
Co
ntr
ibu
tes
to o
rgan
izat
ion
al le
arn
ing
, in
form
ed d
ecis
ion
-mak
ing
an
d
acco
un
tab
ility
fo
r re
sult
s
Qu
alit
y, c
red
ibili
ty a
nd
uti
lity
of
the
eval
uat
ion
s ar
e im
pro
ved
Sup
port
cou
ntry
-led
eval
uatio
n an
d
stra
tegi
cally
pos
ition
UN
ICEF
with
re
gion
al U
nite
d N
atio
ns in
tera
genc
y ev
alua
tions
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–202118
Co
un
try
offi
ce le
vel
Reg
ion
al o
ffice
leve
lIn
dic
ato
rs (
incl
. fr
equ
ency
of
rep
ort
ing
Mea
ns
of
veri
fica
tio
n
Bas
elin
e (r
ef. y
ear
and
so
urc
e)Ta
rget
(re
f. y
ear
and
so
urc
e)
1.
Eva
luat
ion
fu
nct
ion
is s
yste
mat
ical
ly e
mb
edd
ed in
UN
ICE
F’s
resu
lts-
bas
ed m
anag
emen
tC
Os
new
CP
Ds
and
P
SN
s sy
stem
atic
ally
use
ev
iden
ce f
rom
eva
luat
ion
s
RO
sha
res
a co
mpi
latio
n of
less
ons
lear
ned
and
reco
mm
enda
tions
(201
4-20
16)
and
revi
ews
thei
r in
tegr
atio
n in
th
e C
PD
s an
d P
SN
s, S
itAns
Trac
king
of
eval
uativ
e ev
iden
ce u
se in
the
C
PD
s an
d P
SN
s de
velo
pmen
t pr
oces
s (t
hrou
gh a
tem
plat
e to
be
deve
lope
d by
th
e R
O).
CO
CP
Ds,
PS
Ns
and
coun
try
prog
ram
me
resu
lts
fram
ewor
ks a
nd t
rack
ing
repo
rt
Them
atic
eva
luat
ion
reco
mm
enda
tions
inco
rpor
ated
in
the
new
glo
bal s
trat
egy
(e.g
. H
IV) a
nd r
egio
nal s
trat
egie
s (e
.g.
regi
onal
nut
ritio
n st
rate
gy o
r C
4D
stra
tegy
) and
in
4 C
Os
(Cam
bodi
a, C
hina
, In
done
sia
and
Mal
aysi
a) in
201
6 (C
PD
s an
d P
SN
s)
Glo
bal t
hem
atic
ev
alua
tions
fee
d in
to n
ew
RO
MP
Less
ons
lear
ned
and
reco
mm
enda
tions
fr
om e
valu
atio
ns a
re
inco
rpor
ated
in a
ll C
Os
new
CP
Ds,
PS
Ns
and
new
RO
MP
by
2021
CP
D o
utc
om
es a
re
eval
uat
ion
rea
dy
(eva
luab
ility
)
RO
MP
pro
gram
me
outc
omes
ar
e ev
alua
tion
read
y
RO
val
idat
es e
valu
abili
ty o
f C
PD
s
CP
Ds
and
RO
MP
are
be
tter
des
igne
d an
d ev
alua
tion
read
y
All
CO
s an
d th
e R
O
cond
uct
an e
valu
abili
ty
asse
ssm
ent
of C
PD
/R
OM
P t
o be
com
e ev
alua
tion
read
y by
202
1
Whe
n re
ady,
RO
sha
res
HQ
gui
delin
es a
nd p
rovi
des
com
men
ts a
nd t
echn
ical
as
sist
ance
on
eval
uabi
lity
asse
ssm
ent
of C
P a
nd p
ilot
initi
ativ
es
# of
CP
D e
valu
abili
ty
asse
ssm
ents
# of
and
use
of:
-in
depe
nden
t ev
alua
bilit
y as
sess
men
t re
port
-R
O e
valu
abili
ty
asse
ssm
ent-
supp
ort
mis
sion
s
2/14
CO
s : I
ndon
esia
in 2
015
(RO
ev
alua
bilit
y as
sess
men
t-su
ppor
t m
issi
on),
Mal
aysi
a in
201
6 (C
PD
ev
alua
bilit
y as
sess
men
t re
port
)
(eva
luab
ility
ass
essm
ents
pl
anne
d in
201
7) 4
CO
s (M
ongo
lia, T
haila
nd, D
emoc
ratic
P
eopl
e’s
Rep
ublic
of
Kor
ea a
nd
Phi
lippi
nes)
, in
2017
(upc
omin
g ev
alua
bilit
y re
port
)
N/A
to
curr
ent
RO
MP
201
4-20
17
1 R
OM
P e
valu
abili
ty
asse
ssm
ent
by 2
018
https://www.unicef.org/about/execboard/files/2015-PL21-Cambodia_CPD-ODS-EN.pdfhttps://www.unicef.org/about/execboard/files/2015-PL22-China_CPD-ODS-EN.pdfhttps://www.unicef.org/about/execboard/files/2015-PL23-Indonesia_CPD-ODS-EN.pdfhttps://www.unicef.org/about/execboard/files/2015-PL24-Malaysia_CPD-ODS-EN.pdfhttps://icon.unicef.org/apps02/cop/edb/Lists/Evaluation Reports/Attachments/7674/EA Malaysia Final Report (31 August 2016).pdfhttps://icon.unicef.org/apps02/cop/edb/Lists/Evaluation Reports/Attachments/7674/EA Malaysia Final Report (31 August 2016).pdf
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–2021 19
Co
un
try
offi
ce le
vel
Reg
ion
al o
ffice
leve
lIn
dic
ato
rs (
incl
. fr
equ
ency
of
rep
ort
ing
Mea
ns
of
veri
fica
tio
n
Bas
elin
e (r
ef. y
ear
and
so
urc
e)Ta
rget
(re
f. y
ear
and
so
urc
e)
Co
nd
uct
per
iod
ic
trai
nin
g t
o s
tren
gth
en
CO
pro
gra
mm
e st
aff
mem
ber
s’ u
nd
erst
and
ing
o
f th
e ro
le o
f ev
alu
atio
n
and
its
pla
ce in
res
ult
s-b
ased
man
agem
ent
(RB
M)
(ori
enta
tio
n o
f ev
alu
atio
n a
nd
ro
le o
f ev
alu
atio
n in
pla
nn
ing
an
d
imp
lem
enta
tio
n)
Con
duct
per
iodi
c tr
aini
ng t
o st
reng
then
RO
pro
gram
me
staf
f m
embe
rs’ u
nder
stan
ding
of
the
role
of
eval
uatio
n an
d its
pla
ce in
R
BM
(orie
ntat
ion
of e
valu
atio
n an
d ro
le o
f ev
alua
tion
in p
lann
ing
and
impl
emen
tatio
n)
# of
eva
luab
ility
, ev
alua
tion
and
RB
M
trai
ning
s fa
cilit
ated
by
RO
and
CO
s
Cos
and
RO
tra
inin
g re
port
s on
eva
luab
ility
ev
alua
tion
and
RB
M
6 ev
alua
bilit
y as
sess
men
t tr
aini
ngs
(3 C
Os:
Indo
nesi
a,
Phi
lippi
nes,
Vie
t N
am, 1
RO
and
1
trai
ning
in r
egio
nal n
etw
ork
mee
ting
in 2
016)
, 7 e
valu
atio
n ca
paci
ty d
evel
opm
ent
trai
ning
(2
015:
3 C
Os,
1 jo
int
UN
; 201
6: 2
C
Os,
1 R
O, a
nd 1
join
t U
N 2
017:
1
CO
and
1 jo
int
UN
) and
4 R
BM
tr
aini
ngs
(201
6: 1
CO
s, 3
RO
, tr
aini
ngs)
To b
e de
fined
tog
ethe
r w
ith t
he R
O p
lann
ing
unit
2.
Eva
luat
ion
s ar
e p
lan
ned
wit
h a
n a
nn
ual
an
d m
ult
i-an
nu
al h
ori
zon
:A
ll C
Os
eval
uat
ion
s ar
e p
lan
ned
, des
ign
ed a
nd
b
ud
get
ed in
th
e co
un
try
pro
gra
mm
e (C
P)
in t
he
cost
ed e
valu
atio
n p
lan
(C
EP
) an
d IM
EP
or
PR
IME
RO
rev
iew
s C
O c
oste
d ev
alua
tion
plan
s an
d C
PD
and
R
RF
budg
ets
# an
d %
of
CO
s w
ith a
for
mal
cos
ted
eval
uatio
n pl
an a
nd
IME
P o
r P
RIM
E
CE
Ps
and
IME
Ps
or P
RIM
E
Indi
catio
n of
pla
nned
ev
alua
tion
in t
he n
ew
RO
MP
11 C
Os
(Cam
bodi
a, C
hina
, DP
R
Kor
ea, I
ndon
esia
, Lao
PD
R,
Mal
aysi
a, M
ongo
lia, V
iet
Nam
, Th
aila
nd, M
yanm
ar a
nd t
he
Phi
lippi
nes)
hav
e co
mpl
eted
a
cost
ed e
valu
atio
n pl
an a
s pa
rt o
f th
eir
CP
D in
201
7
All
CO
CP
hav
e a
cost
ed e
valu
atio
n pl
an
by 2
019
and
upda
ted
PR
IME
or
IME
Ps
draw
n on
Eva
luat
ion
Pol
icy
pred
efine
d pr
iorit
izat
ion
crite
ria25
and
eng
agem
ent
with
CO
sen
ior
lead
ersh
ip
The
RO
MP
incl
udes
a r
egio
nal
cost
ed e
valu
atio
n pl
an,
spec
ifyin
g m
ulti-
coun
try
and
them
atic
eva
luat
ion
to b
e co
nduc
ted
Exi
sten
ce o
f a
form
al
eval
uatio
n pl
an a
s pa
rt o
f th
e ne
w
RO
MP
Rep
orts
of
RO
sup
port
to
CO
req
uest
sC
oste
d ev
alua
tion
plan
: N/A
to
curr
ent
RO
MP
201
4-20
17A
ll 14
CO
s an
d th
e R
O
have
a c
oste
d ev
alua
tion
plan
by
2018
RO
tim
ely
and
qual
itativ
e ad
vice
on
CO
eva
luat
ion
plan
ning
and
bu
dget
ing
Per
cept
ion
surv
ey
25
Acco
rdin
g to
the
Polic
y, th
e CO
nee
ds to
ens
ure
eval
uatio
n is
und
erta
ken:
bef
ore
prog
ram
me
repl
icat
ion
or s
calin
g up
(e.g
., pi
lot i
nitia
tives
); w
hen
resp
ondi
ng to
maj
or e
mer
genc
ies;
follo
win
g lo
ng p
erio
d of
une
valu
ated
pro
gram
me
impl
emen
tatio
n, e
spec
ially
whe
re th
e pr
ogra
mm
e ha
s be
en im
plem
ente
d fo
r at l
east
five
yea
rs w
ithou
t any
eva
luat
ion
activ
ity; f
or e
ach
outc
ome,
whe
n ex
pend
iture
has
reac
hed
US$1
0 m
illio
n; fo
r eac
h ou
tcom
e, w
here
the
aver
age
annu
al
expe
nditu
re e
xcee
ds U
S$1
mill
ion
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–202120
Co
un
try
offi
ce le
vel
Reg
ion
al o
ffice
leve
lIn
dic
ato
rs (
incl
. fr
equ
ency
of
rep
ort
ing
Mea
ns
of
veri
fica
tio
n
Bas
elin
e (r
ef. y
ear
and
so
urc
e)Ta
rget
(re
f. y
ear
and
so
urc
e)
CM
Ts o
r P
MT
ap
pro
ve
eval
uat
ion
pla
ns
bas
ed o
n
stra
teg
ic a
lign
men
t an
d
reso
urc
es, r
egu
larl
y re
view
th
eir
imp
lem
enta
tio
n
PM
T an
d R
MT
appr
ove
and
regu
larly
rev
iew
reg
iona
l ev
alua
tion
plan
s.
RO
sha
res
less
ons
lear
ned
and
good
pra
ctic
es in
CE
P
deve
lopm
ent
with
CO
s
# of
CO
str
ateg
ic
eval
uatio
ns
# of
mul
ti-co
untr
y st
rate
gic
eval
uatio
ns
in t
he r
egio
n
Min
utes
of
the
CM
Ts,
PM
Ts a
nd R
MTs
CP
DS
, CE
Ps
and
IME
Ps
or
PR
IME
Eva
luat
ion
is a
n ag
enda
item
in:
-4/1
4 C
Os
CM
Ts (m
inut
es)
-PM
T ite
m o
nce
a ye
ar (m
inut
es)
-RM
T ite
m (i
n 20
15 a
nd 2
017,
m
inut
es)
Eva
luat
ion
is a
CM
T or
P
MT
agen
da it
em in
all
CO
s an
d in
one
PM
T an
d R
MT
mee
ting
per
year
A
ll C
O a
nd R
Os
plan
ned
eval
uatio
ns a
re
stra
tegi
cally
cho
sen
by
2019
RO
rev
iew
of
CP
and
C
EP
eff
ectiv
enes
s on
pro
gram
me
impl
emen
tatio
n
RO
MP
and
reg
iona
l ev
alua
tion
repo
rts
repo
rts
of R
O s
uppo
rt t
o C
Os
CM
T e
nsu
res
that
all
CP,
al
l pilo
t p
roje
cts
and
all
pro
gra
mm
e co
mp
on
ents
th
at f
ulfi
l eva
luat
ion
p
rio
riti
zati
on
cri
teri
a se
t in
th
e E
valu
atio
n P
olic
y ar
e ei
ther
rev
iew
ed o
r ev
alu
ated
PM
T en
sure
s th
at e
nsur
es t
hat
RO
MP
and
sec
tor
prog
ram
mat
ic
area
s th
at f
ulfil
eva
luat
ion
prio
ritiz
atio
n cr
iteria
set
in t
he
Eva
luat
ion
Pol
icy
are
eval
uate
d
# of
pro
gram
me
com
pone
nts
eval
uate
d in
the
•
CP
cyc
le p
er C
O•
RO
MP
cycl
e pe
r RO
GE
RO
S a
nd P
RIM
E o
r IM
EP
sA
vera
ge 2
pro
gram
me
com
pone
nts
eval
uate
d pe
r cy
cle
at t
he e
nd o
f 20
16 (G
ER
OS
)
N/A
for
RO
in R
OM
P c
ycle
20
14–2
017
Eas
t A
sia
and
Pac
ific
regi
on d
oes
at le
ast
8 ev
alua
tions
per
yea
r
In la
rge
CO
s, a
t le
ast
five
prog
ram
me
com
pone
nts
are
eval
uate
d in
the
co
urse
of
the
prog
ram
me
cycl
e
In m
ediu
m a
nd s
mal
l CO
s,
at le
ast
thre
e co
mpo
nent
s ar
e ev
alua
ted
At
leas
t tw
o co
re
com
pone
nts
of t
he R
OM
P
are
eval
uate
d by
202
1
3.
Th
e ev
alu
atio
n f
un
ctio
n a
t co
un
try
offi
ce a
nd
reg
ion
al o
ffice
leve
l is
adeq
uat
ely
reso
urc
ed26
Rep
s en
sure
th
at d
edic
ated
h
um
an r
eso
urc
es f
or
eval
uat
ion
are
in p
lace
to
su
pp
ort
th
e ev
alu
atio
n
fun
ctio
n (
CO
)
RD
and
DR
D e
nsur
e th
at
RO
eva
luat
ion
sect
ion
is
adeq
uate
ly r
esou
rced
to
supp
ort
the
eval
uatio
n fu
nctio
n an
d im
plem
ent
regi
onal
str
ateg
y an
d su
ppor
t th
e E
ast
Asi
a an
d P
acifi
c C
Os
# of
CO
s w
ith
adeq
uate
Hum
an
reso
urce
s to
sup
port
ev
alua
tion
func
tions
co
nfirm
ing
requ
isite
st
anda
rds
(CO
and
R
O)(a
nnua
lly)2
7
CO
and
RO
org
anog
ram
s an
d JD
s
EA
PR
O M
&E
foc
al
poin
t lis
t th
at is
upd
ated
qu
arte
rly
3 C
Os
(Cam
bodi
a, M
yanm
ar,
Mal
aysi
a) h
ad o
ne e
valu
atio
n sp
ecia
list
shar
ed p
ost,
9 C
Os
with
PM
E o
r M
&E
spe
cial
ists
, 2
CO
s w
ith M
&E
offi
cers
and
in 1
C
O p
ost
rela
ted
to e
valu
atio
n is
un
der
recr
uitm
ent
at t
he e
nd o
f M
arch
201
7
All
CO
s ha
ve a
dequ
ate
hum
an r
esou
rces
to
supp
ort
eval
uatio
n fu
nctio
ns c
onfir
min
g re
quis
ite s
tand
ards
(CO
an
d R
O)2
8
RO
reg
iona
l eva
luat
ion
advi
ser
P5
and
eval
uatio
n of
ficer
JP
O o
r N
OB
26
The
tota
l cos
t of t
he e
valu
atio
n de
pend
s on
the
com
plex
ity o
f the
pro
gram
me,
the
eval
uatio
n ty
pe, d
esig
n an
d ex
tern
al e
xper
tise
requ
ired.
The
Pol
icy
allo
cate
s a
min
imum
of 1
to 3
per
cen
t of o
vera
ll pr
ogra
mm
e ex
pend
iture
to e
valu
atio
n. 2
7 M
ay in
itial
ly v
ary
acco
rdin
g to
mul
tiple
fact
ors
incl
udin
g si
ze o
f the
pro
gram
me,
ava
ilabi
lity
of R
R an
d OR
28
Hum
an re
sour
ce re
view
s al
tern
ativ
es, i
nclu
ding
sha
red
eval
uatio
n sp
ecia
lists
pos
t opt
ion
acro
ss th
e re
gion
bef
ore
scal
ing
up, t
o be
dis
cuss
ed a
t the
RM
T.
-
UNICEF East Asia and the Pacific Regional Evaluation Strategy and action plan: 2018–2021 21
Co
un
try
offi
ce le
vel
Reg
ion
al o
ffice
leve
lIn
dic
ato
rs (
incl
. fr
equ
ency
of
rep
ort
ing
Mea
ns
of
veri
fica
tio
n
Bas
elin
e (r
ef. y
ear
and
so
urc
e)Ta
rget
(re
f. y
ear
and
so
urc
e)
Rep
s en
sure
th
at r
equ
isit
e
fin
anci
al r
eso
urc
es a
re
ded
icat
ed f
or
eval
uat
ion
to
su
pp
ort
th
e ev
alu
atio
n
fun
ctio
n
RD
and
DR
D e
nsu
re t
hat
re
qu
isit
e fi
nan
cial
res
ou
rces
ar
e d
edic
ated
fo
r ev
alu
atio
n
to s
up
po
rt t
he
eval
uat
ion
fu
nct
ion
an
d im
plem
ent
regi
onal
str
ateg
y an
d su
ppor
t th
e E
ast
Asi
a an
d P
acifi
c C
Os
# of
CO
s us
ing
1%
of t
heir
budg
et f
or
eval
uatio
n an
nual
ly
CO
ann
ual r
epor
t an
d R
O
annu
al r
epor
t
Vis
ion
and
glob
al
dash
boar
d
6 C
Os
used
mor
e th
an 1
% o
f th
eir
budg
et f
or e
valu
atio
n w
hile
R
O u
sed
0.2%
in 2
016
(glo
bal d
ashb
oard
)
As
an a
ggre
gate
, the
Eas
t A
sia
and
Pac
ific
regi
on r
each
ed 0
.7%
in
201
5 an
d 1.
7% o
f its
tot
al b
udge
t ex
pend
iture
for
eva
luat
ion
func
tion
at t
he e
nd o
f 20
16 (S
AP
)
The
Eas
t A
sia
and
Pac
ific
regi
on, i
nclu
ding
CO
and
R
O a
s an
agg
rega
te, u
se
at le
ast
1% o
f an
nual
bu
dget
for
eva
luat
ion
func
tion
on a
n an
nual
ba
sis
All
CO
s an
d R
O u
se a
t le
ast
1% o
f ea
ch a
nnua
l bu
dget
for
eva
luat
ion
by
2021
RO
s us
ed 1
% o
f th
e C
O b
udge
t fo
r ev
alua
tion
annu
ally
29
All
Eas
t A
sia
and
Pac
ific
CO
sen
ior
man
agem
ent
and
PM
E s
taff
or
M&
E s
taff
jo
b d
escr
ipti
on
s in
clu
de
the
eval
uat
ion
fu
nct
ion
.
Whe
n ne
eded
the
RO
can
su
ppor
t C
Os
to r
evie
w jo
b de
scrip
tions
% o
f P
ME
and
M&
E
staf
f jo
b de
scrip
tions
in
clud
e ke
y ta
sks
and
resp
onsi
bilit
ies
to s
uppo
rt t
he C
O
eval
uatio
n fu
nctio
n.
Rev
ised
JD
sN
/AA
ll JD
s of
Sen
ior
man
agem
ent,
PM
E s
taff
or
M&
E s
taff
incl
ude
the
eval
uatio
n fu
nctio
n(20
19)
4.
All
Eas
t A
sia
and
Pac
ific
cou
ntr
y o
ffice
an
d r
egio
nal
offi
ce e
valu
atio
ns
are
adeq
uat
ely
man
aged
an
d q
ual
ity
assu
red
ag
ain
st t
he
UN
EG
No
rms
and
Sta
nd
ard
s.C
Os
dev
elo
p (
An
nex
4)
or
revi
ew t
hei
r ev
alu
atio
n
SO
P
Rol
l out
sta
ndar
d op
erat
iona
l gu
idel
ines
and
pro
cedu
res
for
bett
er e
valu
atio
n pl
anni
ng,
impl
emen
tatio
n, u
se a
nd
diss
emin
atio
n (a
dapt
atio
n of
S
OP
s de
velo
ped
in C
ambo
dia
in
2015
see
Ann
ex 4
)
# of
CO
dev
elop
ing
or a
dapt
ing
eval
uatio
n S
OP
s
% o
f C
O S
OP
s re
view
ed b
y R
O
Act
ual S
OP
sha
red
with
R
O