TERMS OF REFERENCE

For Engagement of an Evaluation Expert

Title of Assignment: EVAL 2017- 03 Evaluation of WIPO’s Partnerships

Name of unit/sector: Evaluation Section, Internal Oversight Division

Place of Assignment: Home based

Expected places of travel (if applicable): Geneva, Switzerland

Expected duration of assignment: August 13 to December 28, 2018, with no renewal possibility

Application deadline: July 25, 2018, 11:59 PM CEST (Geneva time)

1. Objective of the assignment

1. IOD has foreseen contracting one evaluation expert for 48 working days distributed between August 13 and December 28, 2018. The selected expert will undertake the Evaluation of WIPO’s Partnerships in collaboration with the Evaluation Section task manager as defined in this ToRs and the more detailed ToRs included in Annex 1.

2. Deliverables/services and expected outputs

2. Four deliverables are expected out of this evaluation:

(a) Deliverable 1: An inception report

3. The inception report should summarize the desk review, monitoring data, define theory of change evaluation framework, specify the evaluation methodology, detailed evaluation question matrix, list of stakeholders, sampling strategy, data collection instruments including interview protocols and survey templates for internal staff and external stakeholders.

4. As part of this phase, the evaluator will undertake preliminary interviews with WIPO staff and make the necessary arrangements for engaging external stakeholders in the evaluation process.

(b) Deliverable 2: Draft evaluation report

5. The full fledge draft report will follow the UNEG Quality Checklist for Evaluation Reports and the IOD, Evaluation Section evaluation report template. The evaluator will combine quantitative and qualitative research methods.

EVAL 2018-03 2.

6. Data collection tools should include a representative sample of individual staff and external stakeholders’ interviews, workshops. The consultant will need to provide written interview records and stakeholders’ surveys analysis.

7. The evaluator will be in charge of undertaking the survey distribution and analysis will be done in IOD in Survey Monkey. The evaluator will be responsible for arranging meetings for interviews, data gathering and analysis;

(c) Deliverable 3: Final evaluation report

8. Drafting, editing, and publishing of the evaluation report following the UNEG Quality Checklist for Evaluation Report1 and the Evaluation Section report template.

9. The final report should include the following elements:

(a) Executive summary.

(b) Introduction of evaluation, a brief description of the projects, the scope of the evaluation and clear description of the methodology used.

(c) The report should respond to the questions, as define in the ToRs according to evaluation criteria. Moreover, each section of the report should include conclusions and recommendations drawn from the findings assessment.

(d) A matrix of clearly linked findings, conclusions, and recommendations following the evaluation criteria as per ToRs.

10. All deliverables must comply with the UNEG Quality Checklist for Evaluation Reports2, the IOD, Evaluation Section report template, the United Nations Evaluation Group Norms and Standards3 and IOD Evaluation Section guidance documents.

11. All evaluation assessments must be supported by facts and findings, direct or indirect evidence, and well-substantiated logic. It follows that proposed recommendations must be supported by the findings and be relevant, specific, realistic, actionable, and time bound.

12. Depending on the evaluation process, the task manager might assign the external expert with other relevant tasks during the design and implementation of this evaluation. All evaluation products will be issued in English.

3. Description of duties

13. The contracted evaluation expert reports to the evaluation task manager. The evaluator is responsible for the evaluation design, data collection, analysis and reporting as provided in this ToRs. The evaluator will submit a copy edited final report to the WIPO Evaluation Section.

14. The evaluator shall act independently, in line with the UNEG Ethical Guidelines and in her/his capacities and not as a representative of any government or organization that may present a conflict of interest. She/he will have no previous experience working with the initiative or of working in any capacity linked with it.

15. The evaluator should observe the UNEG guidelines, standards, and norms for evaluations in the UN system, as well as the WIPO Evaluation Policy, in the conduct of this assignment. The evaluator needs to integrate human rights and gender equality in evaluations to the extent

1 http://uneval.org/document/detail/607

2 http://uneval.org/document/detail/607

3 http://uneval.org/document/detail/1914

EVAL 2018-03 3.

possible. The evaluator needs to ensure a complete, fair, engaging, unreserved, and unbiased assessment. In case of difficulties, uncertainties, or concern in the conduct of the evaluation, the evaluator needs to report immediately to the evaluation task manager and the Head of the Evaluation Section to seek guidance or clarification.

16. Implementers will support the evaluation by providing desk review documents, contact details of stakeholders as well as any additional documents that the evaluator requests. It will be responsibility of the Program Managers and the Evaluation Section to ensure senior management engagement throughout the process and timely feedback in the quality assurance and factual clarification on the draft and final reports.

17. The Evaluation Section acts as a clearing entity during the main evaluation phases and steps of this evaluation. It prepares the ToRs and selects the evaluator. It reviews the inception report, draft report, performs quality assurance of all deliverables, and outputs produce by the evaluator. Participates in the dissemination of the final report to stakeholders within and outside WIPO. The ES engages WIPO internal stakeholders throughout the evaluation process in supporting the evaluation and validating reports.

4. Timetable

18. The total duration of the evaluation is equivalent to 48 working days and will take place between August 13 and December 28, 2018.

Activity Days

Desk research, mapping and sampling partnerships and study of relevant documentation 7

Preparation of data collection tools and inception report 7

Interviews with WIPO staff and collaborators 2

Interviews with external stakeholders including Geneva based Permanent Missions, Intellectual Property Offices, beneficiaries of WIPO’s initiatives, among other

10

Preparation of case studies 5

Data analysis for draft report write up 8

Preparation and presentation of findings, conclusions and recommendations matrix 2

Revision of report base on comments provided by the ES and Program implementers 2

Final report write up 5

EVAL 2018-03 4.

5. Monitoring and progress control

19. The evaluator will submit the inception report by August 30, 2018.

20. The evaluator will present the preliminary findings, conclusions, and recommendations to the ES and implementers by November 5, 2018 and before submission of the draft report.

21. The first draft report should be presented to the ES and relevant stakeholders for quality assurance and factual corrections at least 3 weeks before the deadline for the submission of the final report. To this end, the report must be presented by November 8, 2018 for quality assurance to the ES before submission to the Program managers. The consultant will revise the report following the comments of the ES and Director IOD before submitting the report to the PMs.

22. The revised draft report will be shared with Program managers (PMs) for comments by November 26, 2018. PMs who will be given sufficient time for the verification of factual findings as well as compliance with the ToRs (approximately 10 working days). Once the consultant has revised the report following the PMs comments. The ES and Director IOD will assess the quality of the evaluation report and the final evaluation report will be shared with the Program managers.

23. The deadline for submission of final report, which factors comments from the PMs, will be December 10, 2018.

24. The contract concludes, and payment is issued upon satisfactory receipt of deliverables and outputs.

6. Required skills and experience

25. The selected expert to work on this evaluation shall have the following expertise:

(a) Education: Advance university degree evaluation, social science, economics, development studies, public administration, or related field.

(b) Experience

(i) At least 10 years of experience in conducting evaluations

(ii) Experience evaluating partnerships and social networks is desirable

(iii) Demonstrated understanding in gender mainstreaming is desirable

(iv) Quantitative and qualitative data analysis expertise;

(v) Ability to conceptualize, analyze, and draw evidence-based conclusions;

(vi) Excellent communication, writing and report presentation skills;

(vii) Fluency in oral and written English is essential. Fluency in another UN language an asset

7. Condition of service

26. The evaluator will serve under the contract for the services of an individual contractor as detailed in the applicable WIPO rules and regulations. The evaluator will not be consider as a staff member or WIPO official but shall abide to the UNEG norms and standards of conduct. WIPO is entitled to all intellectual and other property rights deriving from this exercise.

EVAL 2018-03 5.

8. Payment of the consultancy fee

27. The evaluation consultant’s fee will be paid in line with the following schedule and upon acceptance (part of the quality assurance process) by the ES and Director IOD of the deliverables and outputs mentioned in the ToRs.

(a) Upon acceptance of Inception report: 20%

(b) Upon acceptance of draft report following UNEG and IOD standards: 40%

(c) Upon acceptance of final evaluation report following UNEG and IOD standards: 40%

For consultants outside Geneva, the ES will cover a maximum of two airfares to Geneva in economic class and DSA as per WIPO rules.

9. How to apply?

28. Interested applicants are required to provide the following:

(a) Detailed CV with the name of two references (we will only contact the references of the final candidate);

(b) If possible, a sample of a recent evaluation report.

(c) Provide an indication of your daily fee rate.

29. This is home based consultancy assignment with a maximum of five working missions.

30. Applications with the above details should be sent to julia.engelhardt@wipo.int copying adan.ruizvillalba@wipo.int.

31. The deadline for submitting the application is July 25, 2018, 11:59 PM CEST (Geneva time).

EVAL 2018-03 6.

Internal Oversight Division

Reference: EVAL 2018-03

TERMS OF REFERENCE

Evaluation of WIPO’s Partnerships

July 6, 2018

EVAL 2018-03 7.

TABLE OF CONTENTS

1. INTRODUCTION ............................................................................................................. 9

(A) PARTNERSHIPS CONTEXT IN WIPO ............................................................................ 9

(i) Networking partnerships ............................................................................................. 10 (ii) Coordinating partnerships ....................................................................................... 11 (iii) Cooperating partnerships ........................................................................................ 12 (iv) Collaborating partnerships ...................................................................................... 12

(B) PARTNERSHIP MANAGEMENT .................................................................................. 12

2. PURPOSE, OBJECTIVES, SCOPE, AND QUESTIONS ............................................... 14

(A) PURPOSE AND OBJECTIVES ..................................................................................... 14

(B) SCOPE .......................................................................................................................... 14

(C) CRITERIA AND EVALUATION QUESTIONS ............................................................... 15

3. APPROACH AND METHODOLOGY............................................................................. 16

(A) EVALUATION DESIGN AND REVIEW PHASE ............................................................ 17

(B) DATA COLLECTION PHASE ....................................................................................... 18

(C) REPORTING PHASE .................................................................................................... 18

4. STAKEHOLDER GROUPS ........................................................................................... 19

5. TIMEFRAME AND PROCESS....................................................................................... 20

EVAL 2018-03 8.

ACRONYMS

ABC Accessible Books Consortium

ADR Alternative Dispute Resolution

AMC Arbitration and Mediation Center

ARDI Access to Research for Development and Innovation

ASPI Access to Specialized Patent Information BVGH BIO Ventures for Global Health

CTCN Climate Technology Center and Network

DAC Development Assistance Committee

FIM International Federation of Musicians

FIT Fund-in-Trust

FRAPA Format Recognition and Protection Association

GII Global Innovation Index IGOs Intergovernmental Organizations

IOD Internal Oversight Division

IP Intellectual Property

LDC Least Developed Countries

LL.M Master of Laws

NTDs Neglected Tropical Diseases

OECD Organization for Economic Co-operation and Development

P&B Program & Budget

PCT The Patent Cooperation Treaty

RG Reference Group

SDGs Sustainable Development Goals

TISCs Technology Innovation Support Centers

ToR Terms of Reference

UN United Nations

UNEG United Nations Evaluation Group

UNFCCC United Nations Framework Convention on Climate Change

UNFIP United Nations Fund for International Partnerships

UNOP United Nations Office for Partnerships

USPTO United States Patent and Trademark Office WHO World Health Organization

WIPO World Intellectual Property Organization

WSIS World Summit on the Information Society

WTO World Trade Organization

EVAL 2018-03 9.

1. INTRODUCTION

1. IOD included in its 2018 Oversight Plan the evaluation of the WIPO's Partnerships after a comprehensive risk analysis carried out through relevance, impact, oversight coverage, and strategic priorities of WIPO management and the Member States.

2. The Terms of Reference (ToR) present an overview of the requirements and expectations of the evaluation while providing information on the evaluation's background, objective, scope, and methodology. The Internal Oversight Division (IOD) Evaluation Section under the leadership of Mr. Adan Ruiz Villalba has developed the ToR based on document review and initial consultation with the Global Issues Sector, the External Relations Division and the reference group.

3. The core evaluation team consists of:

(a) Ms. Julia Engelhardt – Senior Evaluator, IOD. She will lead the evaluation;

(b) Ms. Celine Caira, Evaluation Assistant; and

(c) One external evaluation consultant (name will be confirmed after the selection).

4. The evaluation will be conducted between August and December 2018.

(A) PARTNERSHIPS CONTEXT IN WIPO

5. For the purpose of drafting this ToRs, the evaluation section has made used of the OECD/ DAC partners definition, which states that partners are the individuals or organizations that collaborate to achieve mutually, agreed upon objectives. The concept of partnership connotes shared goals, shared responsibility for outcomes, clear accountabilities, and reciprocal obligations. Partners may include United Nations Organizations, governments, civil society, non-governmental organizations, universities, professional and business associations, multilateral organizations, the private sector, among others.4 This definition will need to be further elaborate by WIPO during the inception phase.

6. WIPO’s partnerships are essential to support WIPO in its efforts of improving innovation capacity, exchanging information and tools, advancing on research, increase recognition of the IP system, among others. However, the question remains to answer whether WIPO's programs are delivering better because of the collaboration they have with their partners or despite it. Furthermore, an evaluation of WIPO’s partnerships will serve to strategize and ma partnership activities, determine whether the goals have been met, and provide a learning opportunity to WIPO and its partners. 7. WIPO's partnerships take different forms, and they are not comparable as each fulfills a different objective and are set in different contexts and results might depend on the maturity of the partnership. In general, WIPO cooperates with member states, intergovernmental and non-governmental organizations, and the business sector to help them realize the benefits of the international IP system. However, this evaluation is not intended to assess all WIPO’s partnerships in such broader context but rather sample those partnerships that will fulfil a set of

4 OECD/ Development Assistance Committee (DAC): Glossary of Key Terms in Evaluation and Results Based

Management

EVAL 2018-03 10.

criteria. Here are some examples of criteria that could potentially been applied to sample partnerships:

(a) provide useful information to WIPO`s programs; (b) Have reach necessary maturity levels to be included in an evaluation exercise, for instance: partnerships which have been initiated in 2018 will not be included in this exercise; (c) Partnerships which are in line with WIPO’s strategy goals and purpose; and (d) Partnerships, which have, consider the inclusion of IP gender component.

8. It is to note that these and other criteria to be used for sampling purposes will be further defined with WIPO key stakeholders. 9. Considering that mapping and classification of partnerships was not available in WIPO at the time of drafting this ToRs, the evaluation section has categorized some partnerships by its purpose. 5 However, this is neither a definitive nor an exhaustive representation of all WIPO partnerships. Mapping, categorizing and clustering the various WIPO partnerships will be part of this evaluation. The following categorization will need to be further elaborated during the evaluation inception phase.

(i) Networking partnerships

10. Networking partnerships have the aim of only exchanging information for mutual benefit. Several public-private partnerships (PPPs) seek to encourage multistakeholder engagement and resources in support of agreed policy orientations of the Member States. Some of the WIPO activities undertaken in collaboration with the private sector include:

(a) WIPO Re:Search, which seeks to improve innovation capacity in the area of neglected tropical diseases, malaria and tuberculosis through the sharing of IP and unpublished scientific data between enterprises, universities and research institutions and capacity building. (b) Accessible Books Consortium (ABC), which seeks to assist in the implementation of the Marrakesh Treaty. (c) WIPO Green, which seeks to provide a platform for the exchange of green technologies in support of the goals of the Paris Climate Agreement and related international processes for managing the challenges of climate change and the preservation of the environment. 6 WIPO Green currently has 84 partners from around the globe. For instance, Fujitsu joined WIPO Green marketplace to contribute to the preservation of the global environment by spreading its environmental technologies throughout society. Kopernik collaborates with WIPO to address technology needs in Indonesia and Cambodia.

(d) WIPO Lex partners use this tool as an essential educational and research tool and help WIPO update and enhance WIPO Lex. Their website information resources may contain useful legal resources developed by using WIPO Lex. Hyperlinks to partners' websites are provided as a convenience only and imply neither responsibility for nor approval of, the information contained in those other websites on the part of WIPO.

(e) BVGH Partnership Hub Report - Catalyzing Partnerships for Global Health: The consortium’s objective is to establish partnerships that facilitate sharing of IP assets to

5 Four types of partnerships

5 as defined in the Partnership Evaluation – Guidebook and Resources, Centers for

Disease Control and Prevention, Adapted from Himmelman (1996, 2001, 2002). 6 WO/PBC/25/18 page 21

EVAL 2018-03 11.

advance the discovery and development of new drugs, vaccines, and diagnostics for NTDs, malaria, and tuberculosis.

(f) WIPO Match is an online tool to match seekers of specific intellectual property (IP)-related development needs with potential providers offering resources. WIPO acts as a facilitator of cooperation and publicizes successful matches. The service amplifies WIPO’s resources and multiplies existing partnerships.

(g) The Technology Innovation Support Centers (TISCs), the public-private partnerships of Access to Research for Development and Innovation (ARDI), and Access to Specialized Patent Information (ASPI) provide free-of-charge or low-cost access to a good range of scientific and technological journals and technology databases7 to developing, transition and LDCs.

(ii) Coordinating partnerships

11. Coordinating partnerships have the aim of exchanging information and altering activities for mutual benefit to achieve a common purpose. Some examples of these partnerships include:

(a) Partnerships through the WIPO Coordination Office to the United Nations in New York, with UN Organizations and IGOs to support the implementation of the 2030 Agenda for Sustainable Development (SDGs), the Addis Ababa Action Agenda (AAAA) and other shared global goals;

(b) Partnership with the United Nations Framework Convention on Climate Change (UNFCCC) process, in particular, the UNFCCC Technology Mechanism to facilitate practical collaboration between WIPO GREEN and the UNFCCC Climate Technology Center and Network (CTCN);

(c) The WIPO-WTO-WHO trilateral work on public health, innovation and IP, the UN Inter-agency Task Force on Non-Communicable Diseases, the World Summit on the Information Society (WSIS) and the Internet Governance Forum on bridging the digital divide.

(d) WIPO’s Program on Building Respect for IP maintain close relationships with numerous partner organizations since a wide range of actors influences international and national intellectual property (IP) policies in this area.

(e) The Format Recognition and Protection Association (FRAPA) and WIPO’s Arbitration and Mediation Center (WIPO Center) joined forces in providing alternative dispute resolution services to address problems of format plagiarism or the unauthorized copying of television (TV) formats, such as those used for game, reality or talent shows and sitcoms. Programs using these formats are often remade in different markets using local parties.

(f) WIPO signed an agreement with the International Federation of Musicians (FIM) and the International Federation of Actors (FIA) to support efforts to improve recognition of the significant contributions made by actors and musicians around the world. The agreement seeks, in particular, to help improve the status of performers in developing countries.

7 WO/PBC/25/18 page 16

EVAL 2018-03 12.

(iii) Cooperating partnerships

12. Cooperating partnerships have the aim of exchanging information, altering activities, and sharing resources (e.g., staff, financial, technical) for mutual benefit and to achieve a common purpose. This type of cooperation takes mostly place with the national IP offices.

13. Other activities include the annual Global Innovation Index (GII), published in partnership with INSEAD and the Cornell University Business School, together with some knowledge partners8;

(iv) Collaborating partnerships

14. Collaborating partnerships aim at exchanging information, altering activities, sharing resources (e.g., staff, financial, technical), and enhancing the capacity of another for mutual benefit and to achieve a common purpose.

(a) Funds-In-Trust (FIT) are voluntarily provided to WIPO by donor countries. Such funds are used for technical assistance and capacity-building projects in developing and least developed countries (LDCs), as well as in countries in transition. Financing partners include IP Finland, Japan, Republic of Korea, United States of America/USPTO, and United Nations Office for Partnerships (UNOP)/ UNFIP9 among other.

(b) The WIPO Academy partners with various academic institutions to provide the Master of Laws (LL.M) in Intellectual Property, a joint post-graduate program in intellectual property (IP).

(c) Activities under the South-South and triangular cooperation refer to development cooperation between fellow countries of the Global South. Triangular cooperation includes a developed country, which supports South-South initiatives.

(B) PARTNERSHIP MANAGEMENT

15. All WIPO's substantial programs do have formal or informal collaboration with different partners. WIPO Programs are directly in charge of managing their partnerships. However, as per WIPO’s mandate Program 20 – External Relations, Partnerships and External Offices, which is managed by the External Relations Division, is responsible for ensuring enhanced and more effective engagement of WIPO with the external community, and building partnerships to support the use of IP for development. The Division aims to:

(a) strengthen existing partnerships and developing new and innovative multi-stakeholder partnerships, issues-based alliances and coalitions to support sustainable development and the implementation of the post-2015 Development Agenda;

(b) Promote WIPO's expertise, projects, programs and tools with the UN and other IGOs to achieve greater awareness, recognition, and use of WIPO's services;

(c) Enhance outreach efforts to donors and partners providing support to the WIPO Member States and other WIPO Programs to increase partnership and resource mobilization for IP projects; and

8 As indicated in the WIPO MTSP 2016/21, page 17

9 Program and Budget document 2018/19, Page 39: Funds in Trust Potentially Available for Programming in 2018/19

EVAL 2018-03 13.

(d) In coordination with other Programs, continue to develop institutional arrangements and effective partnerships with non-governmental stakeholders to harness the capacity and expertise of these stakeholders in advancing WIPO's objectives, as well as to enhance their active engagement in WIPO activities.

16. Some of the WIPO business units that have effectively established partnerships included:

(a) The Regional Bureaus play an instrumental role in facilitating the modalities of horizontal cooperation. This includes strengthening partnerships between and among countries and regions, especially in the context of existing sub-regional or regional groupings. The Bureaus ensures coherent planning and reporting of South-South activities at the Organizational level and interface with all internal and external stakeholders.

(b) The WIPO Academy (Program 11) is the core entity in WIPO for training and human capacity-building activities, particularly for developing countries, LDCs and countries in transition. The WIPO Academy partnerships have the objective of promoting national expertise and long-term local capacities, enhancing the impact and extending geographical outreach. The WIPO Academy announced in the 2016/17 Program and Budget document the development of a policy with specific criteria and conditions for partnerships.

(c) The Services for Access to Information and Knowledge (Program 14) provides free-of-charge or low-cost access to commercial patent, scientific, and technical information databases through the Access to Research for Development and Innovation (ARDI) and the Access to Specialized Patent Information (ASPI) programs. Both programs enable IP Offices and academic and research institutions in over 100 eligible the Member States to obtain free or low-cost access to over 20,000 peer-reviewed journals, books, reference works and sophisticated patent search and analysis tools, respectively. ARDI continues to be further integrated within the activities of the Research4Life (R4L) partnership, which includes WHO, FAO and UNEP, in particular as regards the technical back-office administration of the programs, communications, marketing, and training.

(d) Program 14 in coordination with Program 13 (Global Databases) and 15 (Business Solutions for IP offices) assists developing countries in creating digital and machine-readable full text and legal status information of patents through partnerships programs such as LATIPAT (a regional database for Latin American countries) and ARABPAT (a regional database for certain Arab countries).

(e) Program 18 – IP and Global Challenges manages two multi-stakeholder platforms that facilitate effective collaborative networks and technology transfer relevant to health and climate change. Initiatives build on partnerships and collaborations using open innovation structures, networked innovation, and other forms of partnerships to accelerate their impact. Emphasis is also placed on increasing, North-South and South-South collaborations and partnerships that facilitate the global connectivity of developing country innovators. Program 18 also aims to facilitate the diffusion of health and green technologies together with external partners and technology sharing tools.

(f) WIPO Brazil Office enhances efforts, particularly concerning the Global IP Services, through close engagement with government institutions, universities, industry, and other stakeholders that the Office has developed strong ties with; expand the network of stakeholder contacts through new partnerships for the promotion of IP.

(g) The WIPO Office in China responds to rising demand for alternative dispute resolution (ADR) services in China through tailored promotional activities and institutionalized partnerships to encourage the use of the WIPO's Arbitration and Mediation Center (AMC);

EVAL 2018-03 14.

(h) The WIPO Office in the Russian Federation Reinforce and enhance engagement and communication with governmental bodies, academic circles, inventors’ societies and other stakeholders with which the Office has developed strong and productive relationships; create new partnerships, raise awareness and build capacity in the field of IP, particularly among educational and scientific institutions.

A detailed results-based framework for the 2016 - 2019 can be found in Annex 1.

17. Partnerships in WIPO are instrumental in assisting the organization in mitigating some potential risk, as presented in Annex 2. However, its functioning, leverage, support or sustainability has so far never been evaluated.

2. PURPOSE, OBJECTIVES, SCOPE, AND QUESTIONS

(A) PURPOSE AND OBJECTIVES

18. The purposes of this evaluation are to have a better understanding of the dynamics of partnerships, the contextual factors, and to learn from past experiences.

19. This primary evaluation focus will be on:

(a) Map WIPO`s partnerships and assess its maturity levels. (b) Categorize the partnerships considering its purposes and strategies, including those related to IP gender related ones, to better understand and identify lessons to be learnt.

(c) Assess the usefulness of the partnerships and its alignment with WIPO`s strategic goals and expected results, as well as its alignment with WIPO’s gender policies.

20. The evaluation of partnerships will map the current situation with regards strength, weaknesses, opportunities, and threats. In addition to evaluation will assess the degree to which gender equality has been mainstreamed in initiatives which could have a potential impact on the contribution to gender equality.

21. The evaluation results will be used to inform WIPO Director General, ADG Global Issues Sector, and the Director of the External Relations Division, the Reference Group and, other relevant Program Managers and Member States to make evidence-based strategic decisions.

(B) SCOPE

22. The evaluation will assess a sample of partnerships by using specific criteria and categorization which will be defined during the evaluation inception phase. The evaluation scope will be as follows:

(a) Define the concept of partnerships in WIPO and reconstruct the theory of change.

(b) Map the tools and approaches to understand and facilitate decision making about partnerships. Consistency analysis regarding processes and guidance in the preparation of Memoranda of Understandings, Agreements, or any other formal documentation used to initiate a partnership. (c) Map WIPO’s partnerships and assess the partnership maturity or stage of development and framework components. Maturity stages could include: the formation,

EVAL 2018-03 15.

building and maintenance stage, as well as framework components in terms of infrastructure (lead agency, staffing, funding), internal structure (including mission statements, goals, rules and regulations, steering committees, working groups, roles or job descriptions, meetings and communication channels) and processes (decision making, problem solving, training, planning and resources allocation).

(d) Sample partnerships that could be useful for a more in-depth analysis, as part of this evaluation. The sample will depend on the developmental stage of the partnership, its maturity the intended uses and users of the information and questions stakeholders might be seeking answers to, among other criteria WIPO might consider useful to apply in the selection of partnerships to be evaluated.

(e) Identification of the factors that have contributed to the performance and results achievement of the partnership. The identification of results and factors contributing to these results.

(f) Identification of good practices that contributed to the achievement of expected results.

(C) CRITERIA AND EVALUATION QUESTIONS

23. Evaluation is a systematic, objective, and impartial assessment to determine the relevance and fulfillment of broader policy objective and specific targets10, as well as the contribution towards enabling policy influencing. The evaluation team will apply the Development Assistant Committee (DAC) and the United Nations Evaluation Group (UNEG) evaluation criteria of relevance, effectiveness, efficiency and to the extent possible, coordination11.

24. The evaluation will only focus on the sample and provide answers to three aspects of performance:

(a) Relevance:

i. Relevance in relation to WIPO`s strategic goals and expected results and objectives for which they were created.

ii. How are organizations working together? And what are they sharing together?

iii. How do individuals within organizations interact with each other or connected to one another?

iv. To which extent have our partners address IP gender related issues as part of their strategies? and to which extent are they aligned with WIPO’s gender policy and strategic goals?

(b) Effectiveness:

i. Do we have the right mix of partners? And to what extent have partners fulfilled their roles?

10

IOD Evaluation Policy, IOD/EP/2016 11

Organization for Economic Co-operation and Development – Development Assistance Committee (OECD – DAC)

EVAL 2018-03 16.

ii. What is the business model by typology of partnership? And what would be

the lessons to be learnt?

iii. To which extend can WIPO and its partners potentially contribute towards IP gender related issues?

(c) Efficiency:

i. To what extent have the partner organizations achieved more than they could do on their own?

ii. what practices are in place to coordinate the various partnership activities and monitor the results of partnerships included the progress made on IP gender related ones?

iii. Have IP gender related issues been consider in terms of resources?

(d) Sustainability: to what extent are the results that the partnerships contributed to through its interventions sustainable?

25. The evaluation team will elaborate a detailed evaluation questions matrix during the desk review phase.

3. APPROACH AND METHODOLOGY

26. The evaluation will apply a utilization focus approach12And assure, whenever appropriate, the inclusion of key internal and external stakeholders during all phases of the evaluation process. This involvement will be based on suitable methodologies, focusing on interviews, consultations, field missions, meetings, reference group and document reviews.

27. The evaluation team will further elaborate the theory of change, the details of the methodology during the inception phase. It will also define the concept of partnerships in WIPO, the different approaches to partnerships and underpinned the theoretical assumptions and dimensions to elaborate the present ToRs further.

28. The evaluation team will apply mixed methods and concepts such as system thinking, which is useful for understanding complex adaptive systems. Systems thinking will provide more clarity on how things are connected to each other to make the models explicit.

29. To respond to the evaluation questions, the evaluation team will draw at different tools such as factors inventory, integration measures to assess the level of cooperation and integration, partnership self-assessment tool, coalition effectiveness inventory and social network analysis among other. It is expected that the evaluation will also draw on primary and secondary sources of data and involve multiple means of analysis. Furthermore, the evaluation will balance whenever necessary geographical and gender diversity.

12

Utilization-Focused Evaluation (UFE), developed by Michael Quinn Patton, is an approach based on the principle that an evaluation should be judged on its usefulness to its intended users. Therefore evaluations should be planned and conducted in ways that enhance the possible utilization of both the findings and of the process itself to inform decisions and improve performance. Patton, M.Q. (2008). Utilization-focused evaluation, 4th edition. Thousand Oaks, CA: Sage.

EVAL 2018-03 17.

30. To capture different stakeholders perspective the evaluation will undertake a stakeholders analysis. The evaluation will identify who the stakeholders are, where they are and what are their interests. The evaluation will classify stakeholders as follows:

31. During the preparation of the ToRs the following limitation has been identified:

(a) Timing: Program staff might not be available at all times to provide inputs, and external consultations will depend upon the availability of the partners who will determine the adequate time for an interview or consultation.

(b) The limited availability of data might limit an efficiency assessment.

(A) EVALUATION DESIGN AND REVIEW PHASE

32. During this phase, the evaluation will review relevant documentation and undertake preliminary interviews. As part of this exercise the evaluation will:

(a) Review existing policies, guidance material, processes for initiating, monitoring and evaluating a partnership, among other;

(b) Prepare a theory of change for WIPO’s partnerships,

(c) Review relevant documentation including program performance reports, program and budget documents, strategy documents, program reviews, among other;

(d) Prepare an inventory of WIPO’s partnerships, including the extent to which these have been managed and coordinated according to WIPO’s policies and guidelines;

(e) Define the list of internal and external stakeholders and their roles;

(f) Develop the evaluation questions matrix and data collection tools such as questionnaires and interview protocols;

Latents

Keep satisfied

Promoters

Manage closely

Monitor

Apathetics

Keep informed

Defenders

Interest

Po

we

r

Supporters Drivers

Bystanders Blockers

Power of influence

Ag

ree

me

nt

Abstainers

EVAL 2018-03 18.

(B) DATA COLLECTION PHASE

33. During this phase the evaluation will:

(a) Interview key internal and external stakeholders reflecting a diversity of backgrounds according to the nature of the partnership. External stakeholders would include the private sector, universities, beneficiaries, financing and collaborating partners, national IP offices, among others.

(b) Survey partners and measure their level of satisfaction, the effectiveness of activities, the collaborative process. Also, the evaluation might make use of the coalition effectiveness inventory and self-assessment tool to evaluate the strengths and the stage of development of the partnership.

(c) Undertake an in-depth review of a sample of partnerships covering, whenever possible, stakeholder groups, and regional diversity and the extent, these are contributing to the achievement of WIPO’s expected results and strategic goals.

(d) Assess the relevance and effectiveness of the partnerships, and identified any success factor as well as strengths and weaknesses of the partnerships.

(e) Review formal collaboration, coordination, and management practices, including monitoring practices.

(f) Interviews with staff members of the External Relations Division and managing WIPO’s partnerships.

34. The evaluation will include a gender analysis to account as far as possible for gender-related topics.

(C) REPORTING PHASE

35. The evaluation team will prepare an evaluation report following the UNEG Quality Checklist for Evaluation Report13, the IOD, Evaluation Section report template, the United Nations Evaluation Group Norms and Standards14 and IOD Evaluation Section guidance documents.

36. The evaluation assessments will be supported by facts and findings, direct or indirect evidence, and well-substantiated logic. Proposed recommendations will be supported by the findings and conclusions, and be relevant, specific, realistic, actionable, and time-bound.

37. IOD will share the draft Evaluation Report for comments with the primary users of the evaluation namely: Director of the External Relations Division and the ADG Global Issues Sector. For confidentiality purposes, the evaluation report in its draft stage shall not be shared with any other parties outside the ones mentioned in this ToRs unless specified by the Evaluation Section.

38. IOD will make the Final Evaluation Report available for publication, as per the WIPO Oversight Charter. The Director, IOD shall publish the final evaluation report on the WIPO website within 30 days of its issuance. If required to protect security, safety or privacy, the Director, IOD may, at his discretion, withhold a report in its entirety or redact parts of it.

13

http://uneval.org/document/detail/607 14

http://uneval.org/document/detail/1914

EVAL 2018-03 19.

4. STAKEHOLDER GROUPS

39. The evaluation team has prepared a preliminary list of internal and external stakeholders. The team will further refine the list in collaboration with the External Relations Division and other Programs managing or monitoring WIPO’s partnerships.

(a) Primary stakeholders include WIPO Director General, ADG Global Issues Sector, and the Director of the External Relations Division.

(b) The Reference Group (RG,) composed of internal stakeholders. The RG will provide technical input and feedback on the evaluation ToR and during the evaluation process and on the preliminary findings, conclusions, and recommendations.

(c) Secondary stakeholders are the WIPO Programs collaborating with the External Relations Division, external partners, people affected by the partnership activities and potential partners.

(d) Besides, WIPO's partners are also stakeholders in this process, and these include partnerships leaders, organizers, members, funders and people affected by the partnership. A detailed list of stakeholders will be elaborated during the desk research phase.

EVAL 2018-03 20.

5. TIMEFRAME AND PROCESS

40. The evaluation will take place between August and December 2018. The Figure 3 below includes a tentative plan for the upcoming evaluation: Figure 3: Tentative evaluation timeframe15.

July 2018 August 2018

September 2018

October 2018

November 2018

December 2018

January 2019

W1

W2

W3

W4

W1

W2

W3

W4

W1

W2

W3

W4

W1

W2

W3

W4

W1

W2

W3

W4

W1

W2

W3

W4

W1

W2

W3

W4

Phase 1: Evaluation design

Drafting the evaluation, consultants and RG ToRs

ToRs revision

Advertise consultants ToR

Consultants recruitment

Phase 2: Desk research

Define the evaluation framework including the theory of change, activities and monitoring data

Field visits preparations

Preparing detailed evaluation questions matrix and data collection tools and inception report

Translation and publication of surveys

Phase 3: Data collection

Interviews with key stakeholders

Field visits

Surveys sent to stakeholders.

Data analysis

Phase 4: Reporting

Preparation of draft report

The report goes for comments

Report finalization issued

Report publication

15

Dates to be defined following an introductory meeting.

EVAL 2018-03 21.

Annex 1: Results-based framework for Partnerships 2016-2019

PBD Expected Results Performance indicators Program Program

2016-2019

VII.1 IP-based platforms and tools for knowledge transfer, technology adaptation and diffusion from developed to developing countries, particularly least developed countries, to address global challenges

No. of agreements under WIPO Re:Search which lead to new or accelerated R&D in NTDs, malaria, and TB

108 agreements, of which three follow-on agreements (cumulative as at end 2016)

18

Increased no. of WIPO GREEN Members 74 Partners and 808 users (cumulative as at end 2016)

18

No. of agreements catalyzed by WIPO GREEN facilitating knowledge transfer, technology adaptation, transfer and/or diffusion

Data sharing agreements: 9 – Formalized Connections 26 (25 Signed Letters of Intent, 1 Memorandum of Understanding) – Agreements catalyzed: 0 (cumulative as at end 2016)

18

VI.2 Systematic, effective and transparent cooperation and coordination between the work of WIPO and national and international organizations in the field of Building Respect for IP

No. of strategic collaborations with partner organizations on building respect for IP

Seven strategic collaborations (end 2016) 17

2016-2019

VIII.5 WIPO effectively interacts and partners with UN and other IGO processes and negotiations

No. of WIPO-led initiatives in partnership with UN and other IGOs to implementation of the SDGs

Three initiatives (in 2016) 20

WIPO's contributions reflected in UN and IGO reports, resolutions and documents from relevant, targeted processes

20

New joint initiatives with other UN agencies/IGOs Two additional (2016) 21

% of spend through UN cooperation 6.7% (2016) 24

2016-2017

III.2 Enhanced human resource capacities able to deal with the broad range of requirements for the effective use of IP for development in developing countries, LDCs and countries with economies in transition

No. of cooperation agreements and partnerships established in line with the Academy’s new vision

11

2018/19

ABC - number of accessible books produced by local partners/NGOs involved in capacity building projects

3,259 educational titles produced (cumulative as at February 2017)

3

No. of users accessing or downloading WIPO online and print publications, tools and resources for copyright management in the creative industries

TBD 3

2016-2019

III.4 Strengthened cooperation mechanisms and programs tailored to the needs of developing countries, LDCs and countries with economies in transition

No. of national, sub-regional and regional/ interregional cooperation agreements, projects, programs, and partnerships to promote the effective use of the IP systems through sharing of best practices.

9

No. of established partnerships

6 IP partnerships established in 2016 : – 2 MoUs between l’Institut National de la Propriété Industrielle (INPI) (France) and WIPO – 1 MoU between the National Institute of IP of Kazakhstan and the United States Patent and Trademark Office (USPTO) – One agreement between WIPO and the Hellenistic IP Organization (OBI) Concerning the Translation into Greek and Printing of a WIPO Publication (Patents) – 10th WIPO Advanced Intellectual Property Research Forum (Israel) – 2016 Ono Academic College (ONO) (Israel)/ WIPO Seminar

10

Source: P&B document 2016/17 and 2018/19

EVAL 2017-01 22.

Annex 2: Partnerships as an instrument of risk reduction Potential Risks Mitigation strategies Program

Reduced market recognition of WIPO arbitration and mediation services; shifting resource comparisons with existing ADR providers and establishment of further providers.

Leveraging the Center’s position as the international IP ADR specialist: intensified collaboration with other Programs; participation in events through WIPO neutrals or online; further partnering with IP and ADR associations; maximizing the use of the Center’s presence in Singapore and collaboration with WIPO External Offices for increased regional activity; working off new research On user practices and expectations; upgrading of marketing infrastructure.

7

Decrease in UDRP filing, affecting WIPO DNS policy influence and Center status in DNS ADR; pressure on ICANN mechanisms (including from UDRP/URS review) and Center primacy in DNS ADR; challenge to Center case administration and policy development roles; URS use and further provider accreditation causing decrease in UDRP filing; fragmentation of and competition within DNS.

Increasing user-friendliness; adjusting UDRP procedures (where ICANNtolerated); active monitoring of ICANN UDRP/URS review; more partnering with IP and related associations; prioritizing case administration and policy development resources to strike balance between “staying in the market” and Adding specific WIPO value.

7

Decisions taken at the international or national levels result in the loss of stakeholder commitment to actively engage in the implementation of activities that affect long-term plans for enhancing human resource capacities in countries in transition.

Working closely with beneficiaries, donors, and Members States. Planning and coordination of all IP related matters with other international and national organizations and partners.

The decrease of contributions from partner institutions may impact the delivery of the training programs and the establishment of new training courses.

Continuous consultations and coordination with current and prospective new partner institutions.

11

IP content of the training programs of the Academy could become misaligned with emerging global challenges and developments in the IP system.

Continuous assessment and feedback from stakeholders in the Member States as well as training partners, participants, tutors and experts contributing to WIPO Academy activities.

11

Economic or political changes may lead to reprioritizing of focus on the development of national innovation and IP strategies.

Working closely with beneficiaries, donors, and Members States. Planning and coordination of all IP related matters with other international organizations and partners.

30

Countries/stakeholders turning away from WIPO to look towards other international or regional organizations developing activities in the area of building respect for IP.

Leverage partnerships, identify and build synergies with clear, delineated roles between the partners.

17

Lack of commitment of members or partners for WIPO Re:Search and WIPO GREEN could result in insufficient records and thus reduced credibility of the platforms.

WIPO Re:Search: Increase efforts to enhance resources and activities of the Partnership Hub. WIPO GREEN: Ensure that the services offered are aligned with customers’ Needs, effective marketing and maintain close collaboration with existing partners/stakeholders.

18