impact assessment of policy-oriented international agricultural research: evidence and insights from...

World Development Vol. 38, No. 10, pp. 1453–1461, 2010� 2010 Elsevier Ltd. All rights reserved

0305-750X/$ - see front matter

www.elsevier.com/locate/worlddevdoi:10.1016/j.worlddev.2010.06.005

Impact Assessment of Policy-Oriented International

Agricultural Research: Evidence and Insights from Case Studies

TOM WALKERPreviously a Consultant to the former CGIAR Science Council, USA

JIM RYANPreviously a Member of the former CGIAR Science Council, Australia

and

TIM KELLEY *

Secretariat of the CGIAR Independent Science and Partnership Council, Rome, Italy

Summary. — Assessing the impact of policy-oriented research is fraught with methodological difficulties ranging from attributing influ-ence on policy change to constructing an appropriate counterfactual. The six case studies in this special section address these challengesin the context of international agricultural research. Methodological and analytical lessons are drawn about the uptake and influence ofpolicy-oriented research on policy change and about the conduct of cost–benefit analysis for such research. The authors of the studieseffectively used a key-informant approach to document uptake and influence for policy reforms that were believed to have incorporatedwell-defined outputs from policy research.� 2010 Elsevier Ltd. All rights reserved.

Key words — policy research, impact assessment, international agricultural research, influence, cost–benefit analysis

* This special section is dedicated to the memory of Bruce Gardener who

provided much of the intellectual leadership for this initiative. Tragically,

Bruce died in 2008 in the midst of this research project, and his presence

was sorely missed. We are grateful to have the opportunity to celebrate his

memory with this collection of studies. We are also grateful to Carol

Weiss, Robert Paarlberg, and the authors of the case studies for extensive

comments on an earlier draft of this paper. In particular, we thank

Mohamed Ahmed, Jere Behrman, Elisabetta Gotor, Simeon Kaitibie,

Diemuth Pemsl, David Raitzer, and Debbie Templeton. We also acknowl-

edge Derek Byerlee for taking the time to be an honest broker in the

publication of these studies. Two anonymous reviewers performed a

yeoman service in reviewing all the work. Those reviews substantially

improved the final product. Final revision accepted: March 21, 2010.

1. INTRODUCTION

In 1979, Carol Weiss summed up the seemingly dauntingchallenge of evaluating the consequences of policy-oriented re-search:

“Most studies appear to come and go without leaving any discerniblemark on the direction or substance of policy. It probably takes anextraordinary concatenation of circumstances for research to influencepolicy decisions directly (Weiss, 1979, p. 428).”

Assessing the impact of policy-oriented research is fraughtwith methodological difficulties ranging from attributing theinfluence of well-defined research to constructing an appropri-ate counterfactual in cost–benefit analysis (Pardey & Smith,2004). Relatively few studies are successful in making a com-pelling case for the influence of specific research outputs onpolicy change and in persuasively documenting the economiceffects of such change.

Policy-oriented studies in the Consultative Group for Inter-national Agricultural Research (CGIAR) are not immune tothese methodological difficulties. The CGIAR is composedof 15 international agricultural research centers, and it is bestknown for its contributions to the genetic improvement offood crops. Although still grounded in crop improvement,the mandate of the CGIAR has broadened over time. Begin-ning in the late 1970s, policy research has represented anincreasing share of research expenditures in its portfolio, risingfrom 9% in 1995 to about 18% currently. Yet, evidence of im-pacts in the policy arena is scanty and fragile. In contrast, doc-umentation on the impact of research on crop geneticimprovement in the CGIAR is copious and robust (Evenson& Gollin, 2003; Raitzer & Kelley, 2008a).

The seeming contradiction of the increasing resources allo-cated to policy research and the scarcity of transparent and

1453

credible evidence of impact led donors to request an explor-atory study on the consequences of policy-oriented researchin the CGIAR in 2005 (Raitzer & Ryan, 2008). This scopingstudy identified 24 ex post impact assessments in CG centersthat traced the uptake, influence, or impact of outputs frompolicy research. Only three of the 24 attempted to empiricallyestimate the economic impacts of such research. Ten studieswent as far as identifying influence; the balance only assessedthe extent of uptake of policy research outputs. The 24 studiesrepresent less than 4% of the 700 studies conducted to date onthe adoption and impact of CGIAR-related technologies.

The scoping study highlighted the need to commission anumber of additional case studies of the impact of policy-oriented research to augment the evidence and to further thedevelopment of methodologies in this challenging area of im-pact assessment. Eleven Centers submitted 14 proposals for expost impact assessment. Seven proposals were selected in acompetitive grants format. 1 Six of these case studies are

http://dx.doi.org/10.1016/j.worlddev.2010.06.005

1454 WORLD DEVELOPMENT

presented in this special section. 2 They explore the impactpathways whereby research and research-related (e.g., evalua-tion, advocacy) outputs found their way into policy processes,the influence they had on policy change, and where possible,the impacts of associated policy changes on welfare and theshare that could be attributed to policy research. The casestudies are summarized in Table 1.

The components of an assessment of policy-oriented re-search and related activities are usefully itemized as the ele-ments of a pathway from resources committed to a researchproject or program to the policy consequences of the research.These components are laid out sequentially as the “impactpathway,” the main steps of which are: inputs, outputs, dis-semination (or uptake), influence, and impact (SPIA, 2006).In assessing policy-oriented research, two of these steps raisethe most difficult problems of evaluation: influence and im-pact. The studies in this special issue devoted most of theirattention to and had the most difficulty with the influenceand impact stages.

Before documenting methodological and analytical issuesrelated to influence and impact in Sections 3 and 4 of this

Table 1. Summary information on the scope of the six case

Center* Case studylocation

Constraint/problemidentified

Research (relateoutput

BioversityInternational

Global Exercising nationalsovereignty over cropgermplasm, potentiallyrestricting geneflows

“Trusteeship” mgenebanks; faciland honest brokrole; advocacy

CIFOR Indonesia High environmental costsdue to policies andcorruption encouraging largeforest clearing

Research andcommunicationthe political ecothe pulp and pasector, links betfiber sourcing pand natural foreclearance

ICARDA Syria Policies restricting fertilizeruse (allocation) on barley inarid zones

Research showinbenefits of fertilarid zone;recommendationfertilizer applicadifferent zones;policy dialog; ad

IFPRI Mexico Risk of dropping a cashtransfer program deemedeffective at keeping childrenin school longer

Evaluation ofPROGRESA prof conditional ctransfers for effiimpact

ILRI Kenya Colonial dairy policyprotected interests of large-scale dairy producers;criminalized activities ofsmall-scale milk vendors(SSMVs)

SDP produced esupporting policinstitutional refovast numbers ofdepending on dfor livelihood

IRRI Philippines Indiscriminate use ofpesticides on rice withharmful effects on health andecology

Research evideneconomic healthfrom pesticide u

* CIFOR: Center for International Forestry Research; ICARDA: InternationaFood Policy Research Institute; ILRI: International Livestock Research Instit

overview essay, we briefly discuss conceptual definitions andmethods in the next section. The essay ends with a concludingsection that summarizes the findings on the economic benefitsof policy research in the context of international agriculturalresearch.

2. DEFINING CONCEPTS AND APPRAISINGMETHODS 3

Although there exists a huge literature on understandingpolitical action, the art and science of attributing later policydecisions to earlier research outputs is still in its infancy.Defining terms with greater precision is necessary to establisha feasible field of inquiry that strives for methodologicalimprovement. Policy-oriented research is defined as researchthat aims, at least in part, at affecting choices made by govern-ments or other institutions whose decisions are embodied inlaws, regulations, or other activities that generate benefitsand costs for people who are affected by those governmentsor institutions. In the CGIAR, such research is identified first

studies assessing the impact of policy-oriented research.

d) New policy/practiceadopted

Impact described

odel foritationering

An in-trust agreementreached and signed

Crop germplasm flows continue;breeders have easy access tounique sources of genes

exposingnomy ofperweenracticesst

Ministerial decreeadopted requiring millsto source all wood fromplantations by 2009

Improvements in sustainability ofpulp production practices;regulation of the pulp and papersector; and due diligence forforestry investments

gizer in

abouttions ininitiatingvocacy

New fertilizer policy(with credit extended toZone 2) adopted in 1989

Benefits derived from fertilizernow used on barley in Zones 2and 3 in Syria

ogramashcacy and

Mexican governmentcontinued withPROGRESA programin basically the sameform

Faster program implementation;improved program evaluationand project manager training;program continuation beyondpolitical regime changes;spillover to programs in othercountries

videncey andrm, e.g.,SSMV

airying

Revised Kenyan dairypolicy adopted in 2004;training and licensing ofSSMVs

Welfare benefits capturedthrough consumer (via lowermilk prices) and producer surplusas well s to SSMVs

ce on thecosts

se in rice

1992–96 policiesregulating highly toxicinsecticides in rice;training of health officers

Private health cost savingsresulting from pesticide usepolicies estimated

l Center for Agricultural Research in the Dry Areas; IFPRI: Internationalute; IRRI: International Rice Research Institute.

IMPACT ASSESSMENT OF POLICY-ORIENTED INTERNATIONAL AGRICULTURAL RESEARCH 1455

and foremost with social scientists, but physical or biologicalscience projects may also be policy oriented. For example,agronomists undertook research on the responsiveness of dry-land barley yields to fertilizer in order to provide better infor-mation for the Syrian government, which had limited farmeraccess to fertilizer for dryland barley production in the beliefthat such use of fertilizer was unproductive (Ahmed et al., thisissue). According to the view taken in this paper, that researchis properly categorized as policy oriented because the primaryimpact pathway was expected to be via policy changes. In con-trast, crop genetic improvement research generates impact pri-marily via technological changes embodied in new seeds thatare adopted by farmers.

Impact assessment of policy-oriented research can be con-ducted using either supply- or demand-driven approaches. Ifwe start with a policy-related research activity and ask whatits consequences are (the supply-driven approach), any singleproject is unlikely to have identifiable effects because policiesare rarely enacted that are tailored narrowly to a comparableextent. On the other hand, if we start with a policy change andask what role policy research played in generating it (the de-mand-driven approach), the odds for attributing a policychange to specific research outputs from a wide range of alter-native research suppliers arguably could be improved, com-pared to an inquiry that begins with research outputs of asingle supplier and subsequently analyzes their linkages to pol-icy change (the supply-driven approach). Testing the validityof the demand-driven approach, which is akin to focusingon success stories in the impact assessment of agricultural re-search, was one of the central features of this research initia-tive.

(a) The components of the impact pathway

Compared to conventional agricultural research that resultsin technologies, several of the components of the aforemen-tioned impact pathway are similar in character for policy-ori-ented research. Treating costs is largely the same for bothbiological and policy-oriented research. Likewise, standarddisciplinary economic models are available and work wellfor the quantification of economic impact in most contexts re-lated to agricultural research.

But, in contrast to biological research, the stages of output,uptake, and influence can be highly intertwined and not easilytreated in a separable fashion in successful policy research. Inorder to generate influence on policymakers, information frompolicy research must reach them and occupy their attention.Assessing whether or not policy output has gained their atten-tion is problematic. Policy makers have to know about the re-search findings, but that is not sufficient. They also have tobelieve that the output of the research is credible.

Credibility is only one characteristic that enhances the pros-pects for influence. This is a function not only of the marginalpolicy research project such as the six presented here but alsoof the intellectual and professional reputation the institutionconducting the research project has built over the years. Sucha reputational capital represents an international public good,and the international agricultural research centers were able tocapitalize on this effectively in creating influence from the sixpolicy research studies. The IFPRI case study, in particular,illustrates the value of engaging a highly credible and indepen-dent organization to lead an evaluation on behalf of the gov-ernment of Mexico.

A comprehensive review of research impact on education,health, social work, and criminal justice in the United Statesshows that research output that scores well on applicability

and clarity also has a better chance of generating effective im-pact (Walter, Nutley, & Davies, 2003). In the narrower contextof forestry, Spilsbury and Kaimowitz (2002) highlight theimportance of these same characteristics as positive attributesfor the output of policy research. Additionally, they alsounderscore the value of “trialability” and reversibility of pol-icy change to which the research contributes. Trialability hasalso been identified as a desirable trait for improved technolo-gies (Rogers, 1996).

Bayesian methods are a formal means to quantify the degreeto which research uptake changes beliefs (Schimmelpfennigand Norton (2003) and Norton and Alwang (2004)). In thisapproach, there are two steps that analysts of impact assess-ment can usefully investigate: first, evidence that policymakersare paying attention, and second, evidence that after payingattention, their views and hence decisions are influenced bythe research of interest.

Landry, Lamari, and Amara (2003) offer an apt example ofhow information on uptake and influence can be elicited. Theyinterviewed hundreds of officials from Canadian governmentagencies and scored their uptake (the term they use is knowl-edge utilization) from answers to six questions concerningwhether they saw reports of the research, whether they under-stood it, whether they participated in events where the re-search was discussed, whether they had cited the research intheir own reports or other documents, whether they attemptedto use the results of the research, and whether it influenceddecisions in their administrative unit (thus actually incorporat-ing both uptake and influence in their measure). Based on mul-tiple regression of the knowledge utilization score as afunction of characteristics of the research, they found that re-search itself in terms of quality or attempts to meet users needsdid not matter (in the statistical sense) but that the researcher’sefforts to get the findings to the user and the relationship be-tween the user and the research organization were statisticallysignificant. This will not be comforting to academic institu-tions that thrive on research quality as the panacea. It alsosuggests that advocacy may not be a complement to quality re-search in terms of influence and impact. Rather it can be asubstitute and must be used judiciously as it can erode reputa-tional capital.

The impact pathway is at its most complex in the route be-tween assimilated outputs and influence. Statistical analysisaimed at detecting influence in the policy arena (Abelson,2002) and paradigmatic paths of influence pitched at a high le-vel of abstraction (e.g., Sabatier, 1991) are not applicable tothis CGIAR case-study initiative that followed a survey ap-proach of eliciting information from respondents who wereaware of and, in many cases, involved in the process that ledto policy change. Although not used to conduct a formal im-pact assessment of policy-oriented research, a non-probabilis-tic survey approach, focusing on qualitative information, hasbeen used in policy research to examine research uptake andinfluence. Examples in the non-agricultural sector are Hird(2005), Rich (2001), and Weiss (1989).

In the case studies that follow, it appears that the best we cando is to rely on interviews with policymakers and policy ana-lysts supporting them to provide the information that wouldbe more objectively generated by statistical studies relatingmeasures of output to a measure of influence. Instead ofobserving cases where policymakers read or do not read the re-search output and then associating the reading or not with theirvotes, we ask policymakers whether they have knowledge of thepolicy-oriented research output and, if they have, whether itinfluenced their decision or vote. This puts the onus on policy-makers knowing in some detail what they did and why they did


it, and their frankness in providing this information. Thatmakes the approach to policymakers and analysts—questionsasked, the way they are asked, when they are asked — cruciallyimportant.

3. EVIDENCE ON UPTAKE AND INFLUENCE

In this section, we appraise the success and scope forimprovement in using a key-informant approach to elicit reli-able information on uptake and influence from the methodsperspective of the previous section. We also discuss how theresults of the case studies fit into a typology of research utili-zation and what they say about the conditions that enhancethe prospects for the uptake and influence.

(a) The key-informant approach

The documentation of uptake and influence of policy-ori-ented research is the hallmark of these studies. Positive docu-mentation of uptake and influence prepares the way forattribution of subsequent welfare gains. All the authors spentconsiderable time in evaluating the degree to which the policychange of interest could be attributed to the center’s researchor reputational capital. In general, the extra effort that was madeto document uptake and influence via key informant interviewswas worth the time and expenditure. Although some were morepersuasive than others, all the studies reported positive results inthis area. The use of a very knowledgeable single interviewerengaging in structured, interpersonal conversations with keyinformants seemed to provide the firmest evidence on uptakeand influence.

In most studies, the assessments of influence were not con-fined to key informant interviews. Primary and secondary datasources and reports were also canvassed to validate and sup-port information gleaned from key informant interviews.More reliable information on influence also contributed tothe structuring of a more convincing counterfactual. Keyinformants were used to broaden the scope of later interviewsand add other interviewees to the original lists, which is desir-able when a statistically random approach is obviously notfeasible.

The studies as a group suggest that methods for document-ing influence are neither simple nor readily codifiable. The pol-icy process is often a political, multi-actor, multi-interestprocess prolonged over months or even years. Some key infor-mants were closer to the action than others; some played morecentral parts in the development of policy; some had bettermemories of key conversations and critical events; and someleft the arena before final policy decisions were made.

With such complex phenomena under enquiry, simple tabu-lations glossed over the subtleties and nuances of the story, butthey were useful in highlighting contentious areas among keyinformants. For instance, in the ICARDA study, half of thekey informants believed that the policy change to make fertil-izer available for rainfed barley would have happened withoutthe research, albeit considerably later than when it actually oc-curred. For all the other questions, perceptions were morewidely shared on the positive role of the research and advo-cacy effort.

At the other end of the qualitative/quantitative continuum,none of the studies used Bayesian analysis to more rigorouslyprobe the dimensions of uptake that precede influence, asdescribed by Norton and Schimmelpfennig (2001). A Bayesianapproach would have been especially relevant in exploring theelement of surprise (unexpected result in terms of magnitude if

not direction) associated with the research outputs. The ab-sence of Bayesian analysis in applications that seem tailoredfor its use in valuing information, with hindsight, is not so sur-prising, and reinforces Lindner’s evaluation that the Bayesianapproach is difficult to implement (Lindner, 2004). Neverthe-less, without the use of more quantitative techniques, theelement of surprise will be cloaked in uncertainty and itsimportance, a subject of debate as it was for the co-authorsof this paper.

The studies that relied on open-ended conversations seemedto establish more credibility on the issue of influence thanthose that drew solely on written questionnaires, especiallymailed-in responses. The latter may be suitable for perceptionsof impact by key actors, but a written questionnaire approachmay be too constraining to be able to capture the nuances andsubtleties surrounding policy-oriented research influence. TheIFPRI study tried to strike a balance by combining a largersample with questionnaires with a smaller number of keyinformants who were extensively interviewed in a moreopen-ended format. The author of the study also sent tran-scriptions of the survey/interview responses to respondentsto check for accuracy or added thoughts and published theseverbatim, with the respondents’ permission, in the unabridgedversion of the study. This represents good practice and is to beencouraged. The IRRI study featured a round of feedbackfrom the key informants so that they could comment on theperceptions of their colleagues to arrive at a more reliable sub-jective scoring.

Impact assessment lends itself to interdisciplinary research(Ravallion, 2001). Adding a team member with a keen interestand skills in interviewing may help to put uptake- and influ-ence-related research on a firmer base. Incorporating in theteam contextual knowledge of organizational analysis and pol-icy-making processes may also strengthen the case for oragainst attribution and highlight the elements of what maybe a more complex storyline than the tabular information pro-vides.

Economists dominated the elicitation of these aspects in thecase studies. In the future, the documentation of uptake andinfluence could benefit from the expertise of other social andpolitical scientists more familiar with interview techniques,qualitative data analysis, and interpretation and narrativetechniques. Greater disciplinary inclusiveness could improvethe assignment of attribution or contribution and the estab-lishment of appropriate counterfactuals.

More “third-party” evaluations could have been made, butat some cost to information and access. Long-term institu-tional involvement engendered the confidence and respect ofthe key informants that, in turn, made a significant differencein whether they were willing to respond seriously to questionsfrom evaluators.

Overall, the studies use a rather limited range of methods inthe documentation of uptake and influence, but they are ampleto build on and to establish priorities for future work. Onetechnique effectively used was the development of “impactpathways,” or what others have called: “theories of influence.”The authors charted the assumptions about where informationentered the system, the paths it took, and the decision makersit influenced, and then sought to verify these assumptions oftransmission and uptake.

(b) Characterizing the case studies

In The Many Meanings of Research Utilization, Weiss (1979)drew attention to the complexity of the impact pathway andconceptualized research use in the context of the policy-making


process. She identified alternative routes of influence, rangingfrom direct application of research results taken up by policymakers (“knowledge-driven” and “problem-solving” models)to more partial and indirect influence (“interactive,” “political,”and “tactical” models) to cases where research generates find-ings that trickle down very slowly and, in the long run, influencesprevailing opinion (the “enlightenment” model). Although rec-ognizing that the models are overlapping and multi-dimen-sional, the framework itself is useful in considering thepathways of influence (research utilization models) of the sixcase studies in this special section. Although none of the casestudies fall completely into one type of model, the studies arecharacterized as having one or more major elements of severalof Weiss’s constructs.

The knowledge-driven model—the classical description de-rived from the natural sciences—assumes knowledge-drivendevelopment and use. That is, basic research discloses someopportunity that may have relevance for public policy. Ap-plied research is conducted to define and test the findings forpractical action and, if successful, appropriate technologiesare developed to implement the findings and application oc-curs. Although it is possible that some of the policy-orientedresearch conducted in academia and in international agricul-tural research has relevance to this model, 4 none of the sixcase studies follow this utilization pathway.

The problem-solving model is probably the prevailing imag-ery of research utilization. It involves the direct application ofthe research results to a pending decision that policy makerstake. A problem exists and a decision has to be made; informa-tion (or widely shared understanding) is lacking to generate asolution or choose the best one and research provides that.The problem-solving model is thus akin to a demand-led studywhere a specific request for information is made and a study iscommissioned. The expectation is that the research result clar-ifies the situation, reduces uncertainty, and therefore influ-ences the decision that policy makers make.

The Bioversity study exemplifies the problem-solving modelcontext. National sovereignty was threatening internationalcrop germplasm exchange. Bioversity as an international insti-tution identified the problem and played a unique role to helpresolve technical, legal, and political constraints to a solution,through indirect means (including research) and through lead-ing discussions and dialog among national representativeswithin an international forum.

The political model is most characteristic of those situationswhen policy positions and opinions are already hardened, sothe research results are used as ammunition to support apre-determined position, often enhancing the confidence ofdecision makers, reducing their uncertainties, and strengthen-ing their position. The IFPRI case seems to have some of thesecharacteristics, that is, the government needed evidence thatthe existing program was delivering benefits and should there-fore be continued, although slightly modified. The governmentseemed to be pre-disposed to obtaining a positive result, thusconfirming (rather than simply assessing) the value of PROG-RESA. Motivation and pre-disposition differentiate the stud-ies in this sense.

The CIFOR case is another example of research utilizationthrough the political model. This must be seen, however, fromthe perspective of the major NGO players (the primary clientin this case), whose positions had already been staked out inopposition to the paper and pulp producers and processors.

The enlightenment model is, according to Weiss (1979),probably the most common pathway of policy research utiliza-tion, where concepts and theoretical perspectives that researchhas engendered permeate the policy-making process. The

research sensitizes decision makers to new issues and raisesawareness about policy problems or generates findings thatmay run counter to prevailing opinion. Research contributesto a body of evidence that is diffusing through numerous chan-nels and helping shape public opinion and decision makers’perspectives. In this model policy makers rarely are able to citethe findings of a specific study that influenced their decisions,but they believe research has helped shape their ideas and ori-entation (Weiss, 1979). This model is more akin to a supply-led approach where research quality and relevance rather thanadvocacy are the raison d’etre and, as in the case of the knowl-edge model, probably accounts for much of the policy-ori-ented research done in academia and also in the CGIAR.However, with the possible exception of the IFPRI casestudy—where previous IFPRI research on cash transfers andintra-household allocation helped inform aspects of thePROGRESA program—the enlightenment model does notappear to be the pathway of utilization for any of the six casestudies. Indeed, key informants could cite specific researchoutputs that influenced their decisions. This should not beinterpreted to mean that all or even most of the CGIAR’s“successful” policy-oriented research has no relevance forthe enlightenment model utilization pathway. These six casestudies were selected because they had the best perceived po-tential for documenting influence and measuring impact, andnot because they are representative of the most effectively uti-lized or the highest impact policy research outputs of theCGIAR. It may well be the most successful policy-oriented re-search of the CGIAR (with respect to ultimate food securityand poverty alleviation impacts), while not possible to mea-sure (due to indirect, incremental, and circuitous pathways),follows the enlightenment path of research utilization. Assuch, one must be careful in generalizing from the resultsand characteristics of this particular set of successful policy re-search case studies.

(c) Factors conditioning success

Salient issues in the conduct of impact assessment of policy-oriented research were identified in earlier studies (CGIARScience Council, 2006; Raitzer & Ryan, 2008). IFPRI’s workon assessing the impact of policy research has also identifiedseveral conditions that are conducive to the generation ofinfluence and impact (Ryan & Garrett, 2003; Ryan & Garrett,2005). The case studies re-confirmed several of these condi-tions.

The long-term, in-country presence of researchers is onecharacteristic shared by five of the six studies. The case studiesunderscored several of the desirable aspects for establishing alonger term involvement in a well-defined policy setting (Tim-mer, 2004). But it is not just being in the same country formany years; the case studies suggested that good research ismost influential when researchers are interacting closely withcivil society in the policy community, such as NGOs and do-nors. The importance of developing these cross-over relation-ships with non-researchers is epitomized in the CIFOR andILRI cases, where advocacy by such strategic partners playeda large role in contributing to policy change. The country spec-ificity of every case and the desirable attribute of a long-termcountry presence in most of the cases seem to argue for thedecentralization of policy research.

Success was also ascribed to a policy environment conduciveto the assimilation of research results. An enabling policy envi-ronment was underscored in the IRRI case study. The authorsof this study note that if the research on the health costs ofpesticide use had been conducted during the Marcos Regime,


the odds are that the output would have fallen on sterileground. 5 The fall of the Suharto Regime just before CIFOR’sresearch began and the US$14 billion default of the pulpindustry’s largest actor within months of the research’s releasecreated the openness and salience necessary for the effectivedissemination and uptake of research findings.

The role of a policy “champion” was a recurring theme inseveral case studies, including ICARDA’s and ILRI’s. Thechampion was primarily responsible for the continuation ofpolicy-related activities after the research formally ended.

Advocacy figured prominently in almost all the studies.Conflict was also palpable in several of them. For example,ILRI’s case study highlights the so-called “milk wars” duringwhich opposing sides forcefully staked their positions in paidadvertisements, policy forums, and high-level governmentmeetings. The understanding of power relationships clearlymatters in the documentation of the influence of research onpolicy change. That understanding is implicit in several stud-ies.

For a few of the case studies, such as Gotor et al. (thisissue) and Behrman (this issue), the research base seemedthin for the amount of leverage on influence that wasclaimed, and reputational capital was clearly a major elementin allowing Bioversity to be regarded as an honest brokerand in contracting IFPRI to conduct the evaluation ofPROGRESA. The quality of research matters (Templetonet al., this issue). However, being spatially and temporallyin the right place at the right time with substantial effort allo-cated to advocacy seemed more important than the breadth,depth, or quality of research in accounting for impact. Thecase studies as a whole displayed more variation in the latterthan in the former and again reminds us that research qualityand advocacy are not necessarily complements but can besubstitutes.

The inferred absence of a positive linkage between longerterm strategic research and success may be somewhat illusorybecause not all the upstream research inputs were dated andaccounted for in the impact assessments. For example, IFPRIwas contracted as a reviewer for PROGRESA because of theinstitute’s reputation that took many years to accrue (Behr-man, 2007). ICARDA’s farming systems project was basedon several years of fertilizer response work on barley priorto the initiation of the project. Indeed the “honest broker” la-bel that most centers claimed was instrumental in achievinginfluence was itself an accumulation of years of quality re-search and partner interactions that built their general reputa-tion.

4. EVIDENCE OF IMPACT

With regard to the welfare effects of policy change, cost–benefit analysis is the staple methodology for economic ex postimpact assessment. Relevant considerations include a precisedefinition of the intervention of interest, a persuasive counter-factual, and an informative sensitivity analysis of the results(Walker et al., 2008). Although methods-related issues incost–benefit analysis take up the bulk of the discussion in thissection, we also pay some attention to results using estimatedrate-of-return parameters on biological agricultural researchas a comparative benchmark.

(a) Cost–benefit analysis: methods

The case study authors found some areas of cost–benefitanalysis more taxing than others. Constructing a persuasive

counterfactual other than the status quo was the mostchallenging task for assessing the impact of policy change.In particular, the view of policy-oriented research as ameans of hastening the arrival of welfare-enhancing policychange has much appeal as a counterfactual scenario, buttranslating this concept into defensible estimates of the pre-cise time saving for empirical benefit estimation remains achallenge.

Examples of good practice in counterfactual constructionwere also evident in the case studies. Noteworthy in this regardwas the CIFOR case study that established counterfactuals forboth the overall effects of advocacy as well as the contributionof policy-oriented research itself.

The authors of the IFPRI and IRRI case studies were ableto draw on previous research outputs in their impact assess-ments. In contrast, Kaitibie et al. (this issue) at ILRI had tocarry out their assessments from scratch in the sense that theearlier policy research did not contribute empirical resultsfor potential use in the later cost–benefit analysis. Nor didthe literature offer much assistance in valuing difficult-to-mea-sure benefits, with the exception of CIFOR’s case study thatdrew heavily on the environmental literature for benefit esti-mation.

Most of the studies used the mapping of impact pathwayseffectively to identify economic benefits; however, the resolu-tion of how policy interventions generated specific benefitswas blurred in one or two of the case studies, especially thosethat tried to focus on a diffuse mix of policy interventions(Katibie et al., this issue).

Like ex post impact assessments of technology-oriented re-search, most of the studies focused on “young” policies thatrequired the projection of benefits into the future in additionto those already realized. Explicitly identifying realized vs.projected benefits is one example of good practice in this re-gard (Templeton et al., this issue, and Raitzer, this issue),but the threat of subsequent policy reversals, analogous to dis-adoption in technology diffusion, did not warrant much dis-cussion. Lastly, a concentration of “younger” policies madethe documentation of uptake and influence easier becausemore key informants with fresher memories could be assem-bled to opine on the economic benefits and data sources thatcould be used in their estimation.

The majority of the case studies took pains to adopt a con-servative posture in the estimation and presentation of re-sults; this is also to be applauded. But areas for improve-ment were also evident. Some of the studies did not err onthe side of clarity in presenting the findings of their cost–ben-efit analysis nor on the side of transparency in showing theirspreadsheet calculations in the full reports. In general, theauthors could have cast some of their results in partial bud-gets, input–output price ratios, and output–input ratios toconvey information that facilitates order of magnitude calcu-lations.

(b) Cost–benefit analysis: results

One order of magnitude difference stands out: the modelestimated internal rates of return (IRR) on the investmentsin these studies as a group tend to be higher than compara-ble estimates for technology-oriented research. For the fourstudies that estimated them, IRRs ranged between 55%and 65%.

In general, the case studies are consistent with the impres-sion that successful policy research at a country level tendsto be more immediate than equally successful agriculturalresearch such as crop genetic improvement. When all the


relevant factors are in place, the gestation period betweencountry-specific, applied policy research and output availabil-ity for adoption appears to be shorter. The diffusion processfor policy adoption at a national level is markedly differentfrom that for technology diffusion.

Aside from the high estimated internal rates of return oninvestment, another feature peculiar to these policy-relatedcost–benefit appraisals is their relatively small size in termsof net present values (NPVs) compared to success storiesfrom crop genetic improvement. The size of their NPVs ismore akin to natural resource management success storiesthan to technological change from crop genetic improve-ment that seems to have more of an international, multi-country public goods character. Based on estimated NPVsin a true ex post setting, none of these epIAs would rankin the top 20 success stories attributed to CGIAR-related re-search (Raitzer, 2003; Raitzer & Kelley, 2008b). Neverthe-less, the IRRI study demonstrates that the potential forlarge benefits exists if the research focuses on issues of wide-spread economic importance. More than anything, the sizeof the economic benefits is constrained by a single-countrysetting.

Five of the six studies addressed policy impact at the na-tional or sub-national level, but the majority also highlightedthe potential for generating similar policy change in a multi-country setting. Nevertheless, the case studies (with the excep-tion of Bioversity’s contribution to global in-trust agreements)give the impression that policy change is country specific,which may seem surprising to economists who are accustomedto the idea of the location specificity of agricultural technol-ogy, but who are unaccustomed to the notion that there is aneed for more location-specific policies. The perception oflocation specificity mainly stems from the policy adoption pro-cess in the uptake and influence stages and is not necessarilydriven by spatial variation in the consequences of policychange.

5. CONCLUSIONS

At a risk of generalizing from a handful of case studies, andbearing in mind the selective nature of the case studies at hand,two major conclusions can be drawn from this exercise on thedirect attribution and impact assessment of policy-oriented re-search. Documentation of uptake and influence was centerstage in this initiative. Such an emphasis was appropriate be-cause few prior studies on policy-oriented research carriedout by international agricultural research centers were ableto tell a persuasive story of the influence of well-defined re-search on policy change.

The first conclusion pertains to the success or failure of thekey-informant approach in making a persuasive case for theattribution of policy change to research. All the case studies

scored well and indeed exceeded our expectations in docu-menting uptake and influence of policy research. Policy mak-ers have taken steps congruent with what the researchsuggested. The big questions remaining were the ones ofinstitutional attribution. There are other NGOs, governmentagencies, and interest groups involved with the most signifi-cant policy issues, notably in the IRRI and CIFOR cases.In these projects, results from the interviews supported thecase that policy-oriented research was a significant contribu-tor. Therefore, this initiative has resulted in a promising pro-tocol that can be used in an agricultural research context andperhaps even in other contexts.

The second conclusion centers on research resource alloca-tion for policy research relative to other areas that competefor scarce resources in international agricultural research. Bythemselves, the case studies are not a ringing endorsement thatthe CGIAR should invest more heavily in policy research.Although internal rates of return for successful policy researchare as high as or higher than for successful genetic improve-ment research, estimated net present values appear to be smal-ler. Follow-on studies of nested work in other countries areneeded to document the potential for spill-over benefits inthe policy research arena. Spill-overs are indicative of and syn-onymous with the international public goods character of pol-icy research.

Based on earlier case studies described in the introductionand on this set of assessments, the cumulative level of esti-mated economic impact of policy research in the CGIAR isaptly described as modest at this time. It should be notedthat the CGIAR probably has more experience in the con-duct of impact assessment of policy research than most otherinstitutions that invest in policy research in the agriculturalsector. But that experience is still scanty and too episodicand scattered to effectively ascertain whether methodologicalproblems inhibit the ability to demonstrate impact from thealternative hypothesis that the international agricultural re-search system is not yet capable of conducting and organiz-ing policy research that will have impact. More successstories are needed.

Summing up, this impact assessment initiative featuringan emphasis on the documentation of influence and impacthas substantially expanded the number of case studies ofsuccessful policy-oriented research. Now that the concernabout the paucity of such cases in international agriculturalresearch has been highlighted and some good-practice exam-ples have been nurtured and developed, the number of sim-ilar studies forthcoming in the next 5 years should be agood indicator of the impact of this initiative. Five yearsfrom now we should be in a position to determine whetherthese first-generation case studies were the tip of an icebergor the bottom of the barrel. If more case studies do not ap-pear on a regular basis, donors have a right to be con-cerned.

NOTES

1. The call for proposals indicated that either of the two approaches toconducting impact assessment would be acceptable, but with preferencegiven to approach b: (a) Starting with CGIAR projects and working upthe impact pathway, then finally identifying influence and impactattributable to each project (a “supply-driven” approach); (b) startingwith an implemented policy that addresses an issue that has been thesubject of CGIAR research, and working backward along the impactpathway, identifying CGIAR inputs and influence in the policy process

vis-a-vis other suppliers and actors (a “demand-driven” approach). Theproposals were expected to address all steps along the impact pathway,including costs and resources devoted to the research, immediate outputsfrom the research (publications, conferences, and other means ofinformation and advisory dissemination), influence in the policy-makingprocess, and impact of the policies implemented on development goals.Criteria in the ranking of proposals were the rigor, originality, andfeasibility of the research plan to provide credible estimates in the difficult


areas of establishing influence and measuring impacts. The findings of theselected case studies were expected to provide a clear and well-documentedbottom line for the socio-economic value of impacts, a full description andanalysis of the path by which center outputs were used to generateinfluence, and an indication of how that influence contributed to the policyimplemented.

2. The seventh study focused on policy research in inland fisheriesmanagement in Bangladesh (Pemsl & Seidel-Lass, 2009). That studydocumented a rich in-country research experience and was characterizedby the use of social networking techniques to unravel the changes inattitudes in community-based inland fisheries development in a complexmulti-ministerial setting for fisheries policy. Positive attitudinal changewas documented and persuasively attributed to research but changedthinking has yet to result in transparent policy change that could be thesubject of impact assessment.

3. This section draws heavily on and is a synopsis of the uptake andinfluence sections of Gardener (2007) that provided guidance for the casestudy initiative.

4. Studies to understand risk preferences and risk adjustment by poorrural households from which policies are then derived and field tested aresome examples of this model. Indeed, much of IFPRI’s policy researchwould fall into this classification.

5. As a point of contrast, parallel research funded by the RockefellerFoundation on the health costs of pesticide use was also carried by CIPand its partners over several years in a smallholder, intensive potato-growing area in northern Ecuador (Crissman, Antle, & Capalbo, 1998).Similar to IRRI’s work, large adverse effects to insecticide use weredocumented and disseminated. Some success was made in improvingworkers’ safety via the adoption of protective clothing, but, in general, thiswork was not effective in spawning policy change because affordable andtechnically efficient substitutes to insecticides to manage Andean potatoweevil were not available, as was the case with IPM in the IRRI study, andbecause pesticide importers vehemently opposed the regulatory controlsproposed by the research. Government administrators and bureaucratswere aware of the research, but they were not influenced by it because ofthe strength of the pesticide lobby.

REFERENCES

Abelson, D. E. (2002). Do think tanks matter?. Montreal: McGill-QueensUniversity Press.

Behrman, J. R. (2007). Policy-oriented research impact assessment(PORIA) case study on the International Food Policy ResearchInstitute (IFPRI) and the Mexican PROGRESA anti-poverty andhuman resource investment conditional cash transfer program. IFPRIimpact assessment discussion paper no. 27. Washington, DC: Interna-tional Food Policy Research Institute.

CGIAR Science Council (2006). Impact assessment of policy-orientedresearch in the CGIAR: A scoping study report. Rome: Science CouncilSecretariat.

Crissman, C., Antle, J. M., & Capalbo, S. M. (1998). Economic,environmental, and health tradeoffs in agriculture: Pesticides and thesustainability of Andean potato production. Norwell, Massachusetts:Kluwer Academic Publishers.

Evenson, R. E., & Gollin, D. (2003). Assessing the impact of the greenrevolution, 1960 to 2000. Science, 300, 758–762.

Gardener, B. (2007). Methods of assessing policy-oriented research: Areview. University of Maryland, College Park, Maryland, USA:Mimeo.

Hird, J. (2005). Policy analysis for what? The effectiveness of nonpar-tisan policy research organizations. Policy Studies Journal, 33,83–105.

Landry, R., Lamari, M., & Amara, N. (2003). The extent and determi-nants of the utilization of university research in government agencies.Public Administration Review, 63(March/April), 192–205.

Lindner, B. (2004). Second thoughts on a framework for evaluatingagricultural economics research. In P. G. Pardey, & V. H. Smith (Eds.),What’s economics worth? Valuing policy research (pp. 153–173). Bal-timore, MD, USA: Johns Hopkins University Press.

Norton, G., & Alwang, J. (2004). Measuring the benefits of policy-oriented social science research. In P. Pardey, & V. Smith (Eds.),What’s economics worth (pp. 225–251). Baltimore: Johns HopkinsPress.

Norton, G. W., & Schimmelpfennig, D. E. (2001). Using Bayesianapproaches to value policy research. Paper presented at the ministry offoreign affairs, Netherlands–IFPRI workshop on assessing the impactsof policy-oriented social science research, 12–13 November, TheHague, The Netherlands.

Pardey, P. G., & Smith, V. H. (Eds.) (2004). What’s economics worth?Valuing policy research. Baltimore, MD, USA: The Johns HopkinsUniversity Press.

Pemsl, D., & Seidel-Lass, L. (2009). Informal networks in policy processes:The case of community-based fisheries management in Bangladesh.Panang, Malaysia: Mimeo, The WorldFish Center.

Raitzer, D. A. (2003). Benefit–cost meta-analysis of investment in theinternational agricultural research centers of the CGIAR report prepared

on behalf of the CGIAR standing panel on impact assessment. Rome,Italy: CGIAR Science Council Secretariat.

Raitzer, D. A., & Kelley, T. G. (2008a). Assessing the contribution ofimpact assessment to donor decisions for international agriculturalresearch. Research Evaluation, 17(3), 187–199.

Raitzer, D. A., & Kelley, T. G. (2008b). Benefit–cost meta-analysis ofinvestment in the international agricultural research centres of theCGIAR. Agricultural Systems, 96, 108–123.

Raitzer, D. A., & Ryan, J. G. (2008). State-of-the-art in impact assessmentof policy-oriented international agricultural research. Evidence andPolicy, 4(1), 5–30.

Ravallion, M. (2001). The mystery of the vanishing benefits: Anintroduction to impact evaluation. The World Bank Economic Review,15(1), 115–140.

Rich, A. (2001). The politics of expertise in congress and the news media.Social Science Quarterly, 82, 583–601.

Rogers, E. M. (1996). Diffusion of innovations (4th ed.). Ithaca, New York,USA: Free Press.

Ryan, J. G., & Garrett, J. L. (2003). The impact of economic policyresearch: Lessons on attribution and evaluation from IFPRI. Impactassessment discussion paper no. 20. Washington DC: InternationalFood Policy Research Institute.

Ryan, J. G., & Garrett, J. L. (2005). The impact of economic policyresearch: Lessons on attribution and evaluation from IFPRI. In D.Stone, & S. Maxwell (Eds.), Global knowledge networks and interna-tional development: Bridges across boundaries (pp. 37–56). London andNew York: Routledge, ODI and CFPS.

Sabatier, P. (1991). Toward better theories of the policy process. PoliticalScience and Politics, 24(June), 147–156.

Schimmelpfennig, D., & Norton, G. (2003). What is the value ofagricultural economics research? American Journal of AgriculturalEconomics, 85(February), 81–94.

SPIA (2006). State of the art in impact assessment of policy-orientedresearch in the CGIAR. Rome: Mimeo, CGIAR Science Council.

Spilsbury, M. J., & Kaimowitz, D. (2002). Forestry research, innovation,and impact in developing countries – From economic efficiency to thebroader public good. The Forestry Chronicle, 78(January/February),103–107.

Timmer, C. P. (2004). Adding value through policy-oriented research:Reflections of a scholar-practitioner. In P. G. Pardey, & V. H. Smith(Eds.), What’s economics worth? Valuing policy research (pp. 129–152).Baltimore, MD, USA: Johns Hopkins University Press.

Walker, T., Maredia, M., Kelley, T., La Rovere, R., Templeton, D.,Thiele, G., & Doutwaite, B. (2008). Strategic guidance for ex postimpact assessment of agricultural research. Prepared for the standingpanel on impact assessment, CGIAR Science Council. Rome: CGIARScience Council Secretariat.


Walter, I., Nutley, S., & Davies, H. (2003). Research impact: A cross sectorreview. UK: Mimeo, University of St. Andrews.

Weiss, Carol (1979). The many meanings of research utilization. PublicAdministration Review, 39, 426–431.

Weiss, Carol (1989). Congressional committees as users ofanalysis. Journal of Policy Analysis and Management, 8(Summer),411–431.

.sciencedirect.com
Available online at www

impact assessment of policy-oriented international agricultural research: evidence and insights from...

Documents