1

The IEA’s Approach to Assessing the Scientific Quality of CGIAR Research Programs: Illustrated by the PIM Evaluation

Chris Gerrard, Team Leader, PIM EvaluationIAAE Pre-conference workshop on “Measuring the value of policy oriented research”Milan, August 8, 2015

2

The CGIAR’s Independent Evaluation Arrangement (IEA)• Established in 2012, located in FAO, Rome — along side the

Independent Science and Partnership Council Secretariat (ISPC)• Responsible for conducting System-level independent

evaluations• Has now (2015) completed evaluations of five CGIAR Research

Programs (CRPs), with five more ongoing• Available at “http://iea.cgiar.org/evaluations”

• What used to be called CGIAR core funding (mobilized centrally) is now being channeled through 15 multi-Center, multidisciplinary CRPs, one of which is PIM (Policies, Institutions and Markets)

3

What is Policies, Institutions and Markets (PIM)?• Based at IFPRI, involving 12 of the 16 CGIAR Centers• Supports evidence-based research to help improve public

policies and expenditures for pro-poor, sustainable agricultural growth in developing countries

• The CRP with the greatest focus on social science and policy research to achieve the CGIAR’s overall objectives of reducing rural poverty, increasing food security, improving nutrition and health, and more sustainable management of natural resources

• Spent $261 million in 2012–2014, of which $68 million (26%) came from centrally mobilized CGIAR resources

• Was organized into 7 flagships and an eighth cross-cutting flagship on gender, partnerships, and capacity strengthening

4

  Flagship Objectives1 Foresight Modeling Improved prioritization of the global agricultural research effort for

developing countries.

2 Science Policy and Incentives for Innovation

In selected countries of focus, attainment of target levels of investment in agricultural research and rates of return to research that at least meet global averages.

3Adoption of Technology and Sustainable Intensification

Increased adoption of superior technologies and management practices in relevant domains of application.

4 Policy and Public Expenditure

Improved sectoral policy and better public spending for agriculture in agriculturally-dependent developing countries.

5 Value ChainsStrengthened value chains that link producers and consumers with lower transactions costs, increased inclusion of smallholders, and provision of benefits to both women and men.

6 Social Protection Improved design and coverage of social protection programs with particular emphasis on vulnerable rural populations.

7 Natural Resource Property Regimes

Improved use of scientific evidence in decision processes related to sustainability of natural resources important for rural livelihoods.

8Cross-cutting: Gender, Partnerships, and Capacity Strengthening

Strengthened empowerment of women through improved metrics that can be used by agricultural research, development, and policy making to identify needs and track progress in women’s empowerment.

5

The Majority of PIM Research Is Based at IFPRI

CGIARCenter

Principal Investigatorsof PIM Projects

Senior Research Staff Allocating Time to PIM Projects

Number Percent Number Percent

IFPRI 72 52.2% 118 54.9%

ICRISAT 16 11.6% 15 7.0%

Bioversity 9 6.5% 10 4.7%

CIAT 6 4.3% 6 2.8%

IITA 5 3.6% 14 6.5%

CIP 5 3.6% 13 6.0%

ICRAF 5 3.6% 9 4.2%

ILRI 4 2.9% 16 7.4%

WorldFish 3 2.2% 11 5.1%

ICARDA 2 1.4% 2 0.9%

CIMMYT 1 0.7% 1 0.5%

IWMI 1 0.7%

Other 9 6.5%

Total 129 100.0% 215 100.0%

6

The Independent Evaluation of PIM Addressed the Following Issues . . .1. Program Focus: Is it supporting research with clear and coherent

objectives that are responding to global, regional and country development challenges?

2. Inter-Center Collaboration: Is it creating opportunities for researchers to engage in relevant and effective collaborations across Centers?

3. Gender: Is it mainstreaming gender perspectives and analysis in PIM research?

4. Quality of Science5. Partnerships and Impacts: Is it fostering strong and innovative

partnerships for positive development impacts?6. Organizational Performance: Does it have streamlined and

efficient governance and management, with clear accountability?

7

. . . And Followed IEA’s Approach and Methodology for Assessing Quality of Science

Topics Instruments1. Review of previous 

assessments• At the program level• At the activity level

2. Inputs to science quality

• Quality of principal investigators, research staff, facilities, and resources

• Internal quality assurance processes and staff incentives• Staff survey

3. Choice of research topics, design and methods

• Portfolio analysis• In-depth review of selected activities, including three

major agricultural modeling systems• Interviews with professional peers• Partner survey• Qualitative review of 25 randomly selected publications

4. Output quality • Number of publications• Placement of research

5. Impacts of published research

• Citation frequency of PIM publications• Additional metrics

8

Review of Previous Assessments• Barrett et al., Stripe Review of Social Sciences in the CGIAR, 2009.

• The CGIAR was increasingly characterized by short-term planning and reporting cycles that forced researchers to prioritize short-term impacts over sound long-term scientific goals.

• The research agenda had become more fragmented and less focused on the areas in which the CGIAR enjoyed a comparative advantage.

• Disciplinary diversity was lacking as 60 percent of all social scientists in the System were economists.

• Research designs and methods used throughout the research process were deficient in a majority of cases.

• ISPC Commentaries on the original PIM proposal (2011) and on the proposal for extension phase (2014)• The gender perspective is very strong throughout the portfolio.• There is little or no research that focuses on the science-policy interface

that could help inform and improve PIM’s stated impact pathways.• PIM does not have a clear and convincing long-term data strategy.

9

Quality of Principal Investigators, by Peer-Reviewed Publications and Center, 2013-2014

10

Quality of Principal Investigators, by Career

H-Index and Center

11

Reasons Given by Some Researchers for Diminished Productivity • They were managing large and complex projects that involved a great

deal of fieldwork and data collection, which had not yet generated results because of these time-consuming activities.

• They chose to present the results of their research in a non-peer reviewed format to better target policy makers, practitioners and other decision makers in the country where the research was taking place.

• They were doing delivery-type research that sought to provide the research results directly to development actors without delays. It was more difficult to get delivery-type research published in leading journals.

• PIM management had not explicitly indicated expected levels of research outputs. Researchers had to discern this implicitly when their funding was discontinued or new proposals were not approved for funding. PIM management appeared to be using IFPRI-based standards which were historically higher than in their own Centers, which did not have the same quality of research support.

12

CGIAR Staff Perceptions of Quality Assurance Mechanisms, by Center

13

Choice of Research Topics, Design and Methods: Portfolio Analysis of 74 Projects

14

Choice of Research Topics, Design and Methods: Survey of CGIAR Partners

15

Qualitative Review of 25 Randomly Selected Publications

16

In-Depth Review of Three Modeling Systems • Few organizations have a similar combination, range, and

quality of modeling systems as IFPRI, and therefore PIM• Scientists leading these efforts are high quality.• Foresight modeling based on the IMPACT model

• Currently being restructured and enhanced based on an earlier (2010) review

• MIRAGE Trade Analysis Model and MIRAGE Biofuels Model• For analyzing the Doha round of WTO negotiations and the EU

biofuels policy, among other things• Country-Level CGE Modeling

• Widely recognized IFPRI expertise, leading to many requests from countries for analytical support, particularly in Africa

17

Number and Type of Publications, 2012-2014 (both peer- and non-peer-reviewed)Journal article 167Discussion paper 49Book chapter 46Working paper 27Brief 19Project note 14Conference paper 10Project paper 9Book 7Report 7

Factsheet 7Brochure 2Data paper 1Note 1Poster 1Policy paper 1Research report 1Technical guide 1Total 370

18

Placement in Peer-Reviewed JournalsFood Policy 22World Development 17Agricultural Economics 15American Journal of Agricultural Economics 5Journal of Development Economics 4Water International 4China Agricultural Economic Review 3Ecological Economics 3Global Food Security 3Land Use Policy 3

Agricultural and Food Economics 2Agricultural Systems 2Current Opinion in Environmental Sustainability 2Economics Modelling 2Food Security 2International Journal of the Commons 2Journal of Agricultural Economics 2Journal of Development Studies 2Sustainability 2World Bank Economic Review 2

19

Citation Frequency of PIM Publications, Published in 2013

• 58% of PIM papers were cited less than the average frequency for each journal

20

Additional Alternative Metrics• The number of views and their IP location codes• The number of downloads of the paper and IP location• References to PIM research in mass media (to researcher or

paper), classified according to type of outlet• Impact indicators on Researchgate (academic social media

platform)• References to PIM research results on Facebook, Twitter, and

other social media• References to PIM-supported research at high-level

international policy meetings on food security, natural resources, or international development in general.

21

Overall Conclusions• PIM is young program (started in 2012). Therefore, the positive

findings in relation to input quality are more convincing than those relating to output quality and impact, which often relate to legacy research that started before PIM.

• The program is doing well on the relevance of scientific topics and quality assurance mechanisms. It is doing less well in relation to minimum standards of scientific productivity and impact.

• PIM has a good number of very productive researchers whose publications are attracting large numbers of citations.

• IFPRI-based researchers have stronger support systems. The increasing degree of inter-Center collaboration has the potential to raise the scientific quality of social science and policy research at the non-IFPRI Centers.