2015 technopolis group guide to impact assessment of research infrastructures

8/17/2019 2015 Technopolis Group Guide to Impact Assessment of Research Infrastructures

1/22

www.technopolis-group.com

2015

Evaluating and Monitoring the Socio-

Economic Impact of Investment inResearch Infrastructures


2/22

www.technopolis-group.com

Evaluating and Monitoring the Socio-Economic Impact of Investment in ResearchInfrastructures

technopolis |group|, 2015

Elina Griniece, Alasdair Reid and Jelena Angelis


3/22

Table of Contents1. Introduction 1

1.1 Why evaluate the socio-economic impact of research infrastructures? 1

1.2 Previous evaluations of the impact of research infrastructure investment 2

1.3 The aim of this guidance document 3

2. Logical framework for socio-economic impacts of investment in researchinfrastructure 4

2.1 Design and construction phase 4

2.2 Operational phase 7

3. Impact assessment methods and tools 13

3.1 Impacts on economy 13

3.2 Impacts on human resource capacity 14

3.3 Impacts on innovation 14

3.4 Impacts on scientific activity 15

3.5 Impacts on society 15

3.6 IA implementation process, its challenges and limitations 17

Bibliography 18

Table of FiguresFigure 1: Typology of research infrastructures ................................................................ 1

Figure 2: Stakeholders and beneficiaries of investment in research infrastructure ....... 2

Figure 3: Logical framework for socio-economic impact assessment of investment inresearch infrastructure ..................................................................................................... 5

Figure 4: Indicators for economic impacts of RI construction phase ............................. 6

Figure 5: Indicators of innovation impact of RI construction phase .............................. 7

Figure 6: Indicators for economic impact of RI operational phase ................................. 8

Figure 7: Indicators for impact of RI operation on human resource capacity ................ 9

Figure 8: Indicators for impacts on innovation of RI operational phase ...................... 10

Figure 9: Indicators for impact on scientific activity ..................................................... 12

Figure 10: Indicators for impacts on society .................................................................. 13

Figure 11: Implementation of an impact assessment of research infrastructure .......... 16


4/22

Evaluating the socio-economic impact of research infrastructures

© Technopolis Group, 2015 1

1. Introduction

1.1 Why evaluate the socio-economic impact of research infrastructures?

Research infrastructures (RI) refer to facilities, resources (including human) andrelated services needed by the research community to conduct research in anyscientific or technological field. Research infrastructures include:

• Major equipment or group(s) of instruments used for research purposes;

• Permanently attached instruments, managed by the facility operator for the benefit of researchers, industrial partners and society in general;

• Knowledge-based resources such as collections, archives, structuredinformation or systems related to data management, used in scientificresearch;

• Enabling information and communication technology-based (ICT) or ‘e-infrastructures’ such as grid, computing, and software communications;

• Any other entity of a unique nature that is used for scientific research.1

Due to the large number of research communities and complex research needs, thereare very different types of research infrastructures with specific characteristics. Figure1 provides an overarching typology of RIs.

Figure 1: Typology of research infrastructures

Type of researchinfrastructure

Description Examples

Single-site facilityUnified body ofequipment at one physicallocation

High-performance laser system,clean room, coastal observatory,centre of competence

Distributed facility

Network of distributed

instrumentation orcollections, archives andscientific libraries

ELI: European Light

Infrastructure; Council ofEuropean Social Science Data Archives

Mobile facilityMobile vehicles speciallydesigned for scientificresearch

Research vessels, satellite andaircraft observation facilities

Virtual facility

ICT-based system forscientific research,including high-capacitycommunication networksand computing facilities

European Grid ComputingInfrastructure; Digital ResearchInfrastructure for the Arts andHumanities (DARIAH)

Setting-up, or renovating, RI usually requires a considerable level of financialinvestment and a long-term operation strategy. For example, in the environmentalsciences, many research infrastructures bring their scientific return only after decadesof sequential data recording. The investment strategy requires careful planning of theoperation phase and possible future reinvestments. This entails the purchase oftechnologically advanced equipment, clustering of specific skills to enable itsdeployment and devising appropriate governance structures. RIs can also be closelylinked to other research and innovation establishments, such as vocational schools,universities, private and public research centres, research hospitals, businessincubators and science and technology parks.

1 Definition used by the European Commission


5/22



While RIs are designed for research needs, the impacts of these facilities reach beyondfuelling scientific excellence. The advanced technical opportunities and theconcentration of skilled human capital and know-how can foster innovation, createnew or expand the existing markets, attract inward investment, increase economicactivity and potentially have an impact on the social and cultural life in a particular

region. In this regard RIs can be viewed as focal points for continuous interaction between scientific, technological and socio-economic development.2

The construction and operational phase of RIs are largely funded from public budgetseither through national funding from one or several countries, or a mix of national andEU funding. For this reason it is crucial to understand the return on investment in RIto support informed decision-making. Yet it is difficult to quantify and understand thisreturn in conventional commercial terms. Investment in RI brings a broad range of benefits that spreads across wider society rather than serving merely the directstakeholders (owners and users of RI). Official statistics do not sufficiently describethe variety of benefits associated with the development and, more importantly,exploitation of RI. It calls for more elaborate and fine-tuned approaches to account forthe impacts that the RI investment brings on science, economy and society.

Figure 2: Stakeholders and beneficiaries of investment in research infrastructure

1.2 Previous evaluations of the impact of research infrastructure investment

Currently there is no unified framework for the impact assessment (IA) of investmentin research infrastructures. Various conceptual frameworks exist in parallelcomprising a range of observable direct and indirect effects and longer-term impacts.3

The existing studies focus on three types of impacts: 1) the direct and indirecteconomic benefits of spending large amounts of public money in a single location; 2)the industrial knowledge spillovers realised by contractors that design, build andequip research facilities and spin-offs that provide specialist technical services back tothe facility (or other facilities); and 3) the local economic effects and high-technology

2 Rizzuto, C. (2012) Benefits of Research Infrastructures beyond Science, presentation at ERF Workshop“The Socio-Economic Relevance of Research Infrastructures”, 31 May-1 June 2012, Hamburg3 See, for example, project Research Infrastructures: Foresight and Impact (RIFI), project on Evaluation of

Research Infrastructures in Open innovation and research systems (EVARIO), Research InfrastructureGroup of UK Science & Technology Facilities Council, Czech Metodika: Evaluation Methodology forResearch Infrastructures


6/22



clusters that grow up around some of the larger facilities.4 A heterogenous set ofmethods is applied to capture these effects of RI. Most of them address standardeconomic impacts (direct effects) and to some extent economic multipliers.5

Comprehensive and methodologically demanding studies are still rare. Core aspects ofRI benefits, such as their impact on human and social capital formation andinnovation, are not extensively explored. Up to date, the existing literature providesinsufficient evidence to support claims that investment in RI (even large-scale) attractand retain talent and promote innovation.6 Therefore, important socio-economiccontributions remain poorly understood. In this respect, significant work isundertaken by FP7 project “Research Infrastructures: Foresight and Impact (RIFI)”. AForesight Enriched Research Infrastructure Impact Assessment Methodology(FenRIAM)7 was developed during the project, which significantly contributed to theunderstanding of the impact of RI on learning and capacity of RI operators, suppliersand users. A key lesson is that there is a need to treat the impact pathways in greaterdetail and substantiate them with empirical evidence.

Most previous concept papers and evaluations have been devoted to studying large-scale research infrastructures, or so-called ‘big science’ facilities that are of European

and international significance. Empirical work on investment in mid-sizeinstrumentation of national and macro-regional significance barely exists. This ismainly the result of the fact that RI investments have not yet been subject to regularevaluations. There is a scope for further methodological developments and theirapplications in evaluations to evolve from narratives of socio-economic effect of RIs tomore empirical models and measurement techniques. This knowledge would help tomove from simple ex-post detection of intended and unintended returns of RIs to a better understanding and planning of future investment.

1.3 The aim of this guidance document

The aim of this guidance document is to disentangle and characterise the concrete benefits of RI investment for different stakeholders and to build a schematic impactassessment framework that can be used in evaluations to trace the core impactpathways. Section two introduces a logical framework for the impact assessment of RI.Recognising that each RI is embedded in specific socio-economic conditions, there willnever be a ‘one-size fits all’ approach to mapping all socio-economic impacts. This work should be regarded as a step towards defining a typology of possible effects andaccounting for the conditions and patterns that enable their creation and diffusion.

The third section elaborates on the methods and tools that are suitable for themeasurement of respective impacts. The identified impact pathways involve complexand multidimensional phenomena, which in most cases cannot be captured withstandard quantitative indicators, but require a thorough triangulation of quantitativeand qualitative evidence. The section elaborates on the appropriate indicators for eachtype of impact and provides additional guidance on the respective data requirements,their potential sources and advice on collection routines. Reflections on the limitation

and challenges of the application of this IA framework are outlined in conclusion.

4 Simmonds, P. et.al (2013) Big Science and Innovation. Report for the UK Department for Business,Innovation & Skills

5 Multipliers - Further economic activity (e.g. jobs, expenditure or income) associated with the inputs to oroutputs from the project. According the European Commission (DG REGIO) an income multiplier is asecondary effect resulting from increased income and consumption generated by the public intervention(investment). Multiplier effects are cumulative and take into account the fact that part of the incomegenerated is spent again and generates other income, and so on in several successive cycles. In each cycle,the multiplier effect diminishes due to purchases outside the territory. The effect decreases much faster

when the territory is small and when its economy is open.6 Horlings, E. et.al (2012) The societal footprint of big science. Report of the Rathenau Instituut, the Hague,

the Netherlands7 http://www.fenriam.eu/


7/22



2. Logical framework for socio-economic impacts of investment inresearch infrastructure

Figure 3 provides a schematic overview of the various direct and indirect benefits that

arise during two main periods of a research facility’s lifecycle:1) design and construction phase; and

2) operational phase of research facilities.

The specificities of each of the pathways and their contributions to other impacts thatare singled-out in this two-dimensional graph are described in detail in the followingsub-sections of this chapter. The types of impacts are positioned in the graph to visually reflect better their relation to each other, not an order of their importance.Societal impacts can be of very diverse nature and lead to broad, overarchingoutcomes. In order to provide a concise graphical representation of IA frameworkthese impacts are not schematically reflected in this graph.

This IA framework is predominantly designed for single-sited and geographically

confined national or macro-regional distributed research infrastructures. With regardsto highly distributed and virtual RI case-specific a range of additional factors have to be considered. These types of RI usually are decentralised administratively andfinancially and include much larger geographical areas of impact. They can havemainly or only virtual access without the requirement to cover physical support toindividual users. Advanced ICT-based technical solutions and well-designed accessrules may ensure much larger outreach with dispersed impact on user and stakeholdergroups. These characteristics of highly distributed and virtual RI require more case-specific approach to impact assessment.

2.1 Design and construction phase

2.1.1 Economic impacts

During the design and construction phase a wide range of technical and scientificknowledge is applied to set-up the required facilities. The physical construction andrefurbishment of buildings and their accompanying networks and utilities engagesmany companies ranging from construction material suppliers to technical and legalservice providers. The procurement of core construction services typically involveslocal companies8, which creates additional revenue and increases temporaryemployment opportunities in the region. Despite the fact that the impact of many ofthese expenditures is usually confined and short-lived for smaller-scale RI, there can be a significant further multiplier effect on the local economy, for example, dueimproved reputation credentials for firms that supply high-tech facilities and anincrease in the local tax base. In-depth analysis of the local industrial base andmapping of relevant global supply chains is required to scope the total economic effectof the RI investment.

8 Locally-owned businesses, franchise or corporate branch operating within a local area


8/22

Evaluating the socio-economic impact of research infrastructure


Figure 3: Logical framework for socio-economic impact assessment of investment in research infrastructure


9/22



Figure 4: Indicators for economic impacts of RI construction phase

No Indicators Explanatory remarks and examples

1. Number of commercial suppliersfor RI design and construction

phase

Suppliers can be mapped per sector, fieldof activity, size, technology group (low-

tech/high-tech), geographic scope(national/international/multinational)

2. Scale of commercial suppliers’turnover increase due to RI

For applied method to estimate utilities inmonetary terms see the study “Economic

utility resulting from CERN contracts”9

3. Scale of commercial suppliers’employment increase due to RI

Measured in FTE and grouped percategory of employees (e.g. technical staff,R&D staff)

2.1.2 Impacts on innovation

Research facilities under construction may require specific design and buildingsolutions to serve the purposes of a particular research field (e.g. laboratories, cleanrooms, etc.). Such requirements foster collaboration among the involved suppliers,facility mangers and scientists in developing innovative design solutions and buildingfunctionalities. Local companies can benefit from accumulated know-how andcomplementary skills that expand their competitive advantage in other markets (e.g.enable winning international procurements of similar research facilities, secure servicecontracts as external consultants to other developers). In cases where foreigncompanies demonstrate a superior skills base and are selected to address particulardesign and construction tasks, the benefits of the investment flow out of the nationaleconomy. Moreover, in the RI construction phase, the universities and responsiblegovernment agencies may acquire additional and more sophisticated projectmanagement skills that can have knock-on effects on the potential for public sectorinnovation.

The purchase of the necessary scientific instruments and supporting equipment isgenerally open to international bidders, as the supplier base for such specificcommodities and accompanying services is often very specialised and, in certain cases,the specific equipment may not exist ‘off-the-shelf’ on the market. Hence, theprocurement of equipment can have a considerable impact on extending thetechnological frontier and fostering innovation10. As a result, the impact from thisprocurement will be distributed across specific global supply chains for scientificinstrumentation.

9 Bianchi-Streit, M. et al (1984) Economic utility resulting from CERN contracts (second study). Report ofthe European Organisation for Nuclear Research (CERN)http://indico.cern.ch/event/66952/contribution/0/2/material/paper/0.pdf

10 For the theoretical background of this nexus see work of von Hippel, E. (1976) The Dominant Role ofUsers in the Scientific Instrument Innovation Process, Research Policy 5 (3)


10/22



Figure 5: Indicators of innovation impact of RI construction phase


1. Number of jointdevelopment activities

with suppliers

The novel technical requirements for theconstruction (design or building techniques) of

scientific facilities or the equipment requireclose interaction between the recipientorganisation and one or more suppliers. Thiscan be realised through a process of pre-

commercial public procurement11, designcontest, a forward commitment procurement ora competitive dialogue. In these cases, therecipient organisation defines specific technicalrequirements or criteria to be met but not theprecise solution (in the form of a specific pieceof equipment, etc.).

2. Number of contracts

concluded for high-tech orspecialist services thatrequire development, orcalibration ofdesigns/equipment tomeet specific requirements

The focus should be on the extent to which

specific design or technical requirements forscientific buildings or equipment induce animpact on learning and skill developmentamongst supplier firms, notably local (national)firms.

2.2 Operational phase

Two main categories of impacts of the operational phase of RI exist: 1) impacts fromthe routine operation, maintenance and upgrading of RI; and 2) impacts from the useof research facilities.

2.2.1 Economic impacts

Benefits from RI operation include the assessment of the performance characteristicsof the research facilities. This includes a review of RI performance addressing aspectsof quality, access, reliability of the infrastructure, account of all the services providedand assessment of the optimal usage of equipment. Among the criteria to review is theflexibility of staff to support external users, feasibility of contractual arrangements forgranting access, quality of accompanying services (e.g. tech transfer, incubationservices). Performance aspects have a direct impact on the efficiency and effectivenessof RI operations, user satisfaction, and consequently on the generated revenues to RIand its users.

Maintenance and operation of RI involves longer-term effects on employment in

higher education establishments, science parks and companies, such as additional jobsfor scientists, technicians, administrative and support personnel working on the RI.The operation of facilities also includes expenditure on goods and services and routineupgrades with additional procurement design and associated equipment. Theseactivities have a further multiplier effect on the local economy and relevant globalsupply chains.

11 http://www.innovatiefaanbesteden.be/files/409


11/22



Figure 6: Indicators for economic impact of RI operational phase


1. Number of scientists,students, state-owned orprivate enterprises that benefitted from RI services

Service can be categorised into broad types,such as: free or for a fee access to equipment(including technical support from qualifiedtechnicians), testing or metrology (carried out by the RI staff in laboratories or using mobileequipment), contract research, consultancy,hosting doctoral students, etc.

2. Total amount of fundinggenerated from services,grants and joint projects

Per service type, source of research grant andtype of joint project

3. Number of new directlyand indirectly created jobs

Per category (scientific/ technical/administrative) and wage level; the aim is to

scope how RI impacts on an increasedemployability of human resources.

4. Total amount ofexpenditure on personnel,operations, maintenance

For centralised large-scale facilities the rule ofthumb is to calculate around 10% of investmentcosts for annual operation expenditure. Yet formost other types of RI this percentage issignificantly higher.12 For distributed RI thereare higher costs due to coordination,management and consultation tasks requiringqualified personnel. Operating costs for RI inthe humanities and social sciences are usuallythe most significant expenditure.13

5. Total RI capacity

utilisation

RI access hours used as % of the total available

access time6. RI capacity utilisation

external business usersRI access hours used by business entities as % ofthe total available access time

7. Financial sustainability ofRI

Measured as % of the total costs funded fromthe provided services, received grants andrealised joint projects

2.2.2 Impacts on human resource capacity

The build up of human resource capital is a major benefit of RI. Modern researchfacilities can serve as magnets to retain and attract talent. This includes localresearchers, technicians, students and global talent through recruitment of permanentstaff and the promotion of the flow of visiting researchers and foreign students.

RI has a crucial role to play in training and skills development. Research facilitiesconcentrate skilled staff that hold tacit knowledge how to operate them. Only with thehelp of skilled technicians and experienced researchers that students and otherinterested stakeholders can learn to set up experiments and interpret their results.This tacit dimension of knowledge transfer makes RIs entry points into networks ofknowledge, expertise and practice. To determine the impact of RI on skillsdevelopment it is important to explore how the exchange between the younger staff

12 Wissenschaftsrat (2013) Report on the Science-Driven Evaluation of Large Research InfrastructureProjects for the National Roadmap (Pilot Phase)

13 Wissenschaftsrat (2011) Recommendations on Research Infrastructures in Humanities and SocialSciences


12/22



(MAs, PhDs, post-docs) and experienced researchers and technicians is structured andpromoted. Equally important is to appraise the ways tacit knowledge is transferred toprivate sector users.

The diffusion of knowledge gained by the RI management, operators and users mayhave a significant societal value. For example, the research commercialisation skillsacquired by the management and scientists at a RI is an important benefit with far-reaching implications on the local innovation system. Closer formal and informalsocial networks increase interpersonal trust and knowledge sharing. This consequentlyaugments the quantity and diversity of knowledge that is available to RI users. Socialnetworks create new forms of interaction among the actors of the innovation system,stimulate learning environments and increase the awareness of users (potential andexisting) about the scope of knowledge that is available on RI. In short, social capitalenables the full exploitation of accumulated human capital.

The mobility of people and emergence of new cooperation networks is a majormechanism through which knowledge circulates and diffuses the benefits to widereconomy. Graduates leave RI with the knowledge of the most recent scientific results,skills using advanced instrumentation and ability to apply them to complex problem

solving. Internships and recruitment of graduates mean that students act as a form of‘social glue’ and they often create long-term links between scientists and businesses.14 However, if the local academic and industrial environment does not provide sufficientabsorption capacity for these advanced skills, the accumulated human capital mayremain underutilised or emigrate.

Figure 7: Indicators for impact of RI operation on human resource capacity


1. Number of new jobs forresearch and technical staffattracted from abroad as %of the total number of staff

employed on RI

Per category (permanent/visiting) and per year;the aim is to understand the influx of newknowledge that is directly linked with RI

2. Number of Master thesisdefended, whereknowledge and skillsgained on RI wereexploited

Per year of completion and per scientific field;Master theses are considered as atraining/education impact on skills profile ofpopulation; the number of PhD thesiscompleted is counted in the impacts onscientific activity

3. Number of graduatestrained on RI

Per stage (MA, PhD) and per year; distinctionshould be made on skills gained by PhDstudents that improve their general academicand employability profile and skills directlyapplied in PhD thesis work; the latter shouldaccount for impacts on scientific activity

4. Number of foreignstudents as % of allstudents trained on RI

Per stage (MA, PhD) and per year

5. Data on the post-diplomaemployment path of thosegraduates trained on RI

Descriptive data should include at leastindication on the share of graduates findingemployment in academia/public sector/privatesector; further statistics on types of graduateemployment (part time/full time) and sector ofactivity are desirable

14 Bozeman, B. (2000) Technology transfer and public policy: a review of research and theory. ResearchPolicy, 29(4–5)


13/22



2.2.3 Impacts on innovation

Innovation can result from RI through industry collaborations with academic researchgroups and contract research that result in knowledge transfer. Research facilitiesgenerate knowledge that businesses cannot acquire through their networks. Moreexposure to RI knowledge networks through formal and informal social interactionincreases the possibility that private sector discerns new opportunities for exploitingintellectual and technical capital available on RI.

Spin-offs are one way in which knowledge generated using RI can diffuse into themarket in the form of innovative product, process or service. Other avenues forinnovation impacts arise from the commercialisation of research through licences or joint ventures with existing companies.

There are important barriers for private sector use of RI due to different objectives and‘cultural factors’. The industrial community is often less interested in using RI directly(e.g. placing industrial researchers in RIs to carry out R&D). There is more interest in

acquiring technical training and co-developing new IPR.15 In some sectors (like food oragriculture) the industrial landscape can be dominated by a large number of SMEs

that, individually, lack resources to engage in research collaboration.The table below provides a number of indicators that can be used to track the impactof RIs on knowledge transfer and innovation. One example of a framework formeasuring the quantity of knowledge transfer from the research base is the Scottish

Funding Council's (SFC) Knowledge Transfer Metrics Return16.

Figure 8: Indicators for impacts on innovation of RI operational phase


1. Number of collaborativeresearch projects and volume of funding

Collaborative research carried out at that the RImight involve both regional and internationalresearchers using the equipment for shorter-or-longer periods. The funding generated from

service fees charged to research team will enablereinvestment in equipment. Qualitative researchon the research teams working in or with the RIcan be used to map (e.g. using social networkanalysis) the place of the RI in global researchnetworks.

2. Number of R&D projectscommissioned bycompanies and volume oftheir funding

Contract research is one step closer togenerating a socio-economic impact since firmspurchasing research services will only do so inorder to improve production processes(enhancing productivity) or develop prototypesof new products or services. Tracking trends in volume of funding provides an indicator of the

longer-term financial sustainability of the RIs.Similarly, tracking the renewal of the ‘client base’ (contracts with new clients) rather than just number of companies contracting researchis useful.

15 ERF Workshop “The Socio-Economic Relevance of Research Infrastructures”, 31 May – 1 June 2012,Hamburg, Germany

16 The metric is used by the Scottish Government to measure as a proxy measure of the quantity, but not thequality, of knowledge exchange activities undertaken by Scottish universities.http://www.gov.scot/About/Performance/scotPerforms/TechNotes/knowledge


14/22




3. Number of technologies,prototypes, industrialdesigns developed and

transferred

Depending on the RI’s intellectual policy policy,the results of collaborative and contractresearch may be potentially exploited through

proof of concept and prototyping either by theRI itself or in co-operation with academic orindustrial partners. Ideally, the monitoring ofoutcomes should go beyond ‘number counting’and aim to track whether prototypes have beenscaled up or further developed commercially.

4. Number of start-ups andspin-offs created withsupport from RI services –growth in turnover/valueadded and employment

The number of start-ups/spin-offs is anindicator commonly used to assess economicimpact. However, there is a need to avoidunrealistic expectations when setting targets fornumbers of spin-offs as this involves IP transferto the new companies potentially closing futureavenues of academic research. Moreover, many

spin-offs fail to grow substantially and there isan need to track growth in turnover andemployment on a longitudinal basis (over 5-10 years) to fully understand impact.

5. Number of feasibility ormarket studies forindustrial investment andapplication of technologies

Commercialisation of research results willinvolve, traditionally, market research orindustrial scale feasibility study prior toinvestment. The number of (e.g. proof ofconcept/prototyping) projects that move to thestage of industrial investment is a strongindication of the economic relevance of theresearch. Tracking over time (5-10 years) theactual investment in the application of new

technologies by businesses provides a strongerindication of economic impact.6. Procurement contracts

signed for developmentand upgrade of researchequipment

Tracking the number and type of procurementcontract for the development of innovativeinstruments and products (rather than simply‘off-the-shelf purchases from equipmentsuppliers provides an indication of the impact ofthe RI on the scientific equipment market – thisis particularly pertinent if suppliers areregionally based.

2.2.4 Impacts on scientific activity

The impact of RI on scientific activities can result in accumulation of new knowledge

and methodologies to push the boundaries of fundamental science. There is aninherent tension between the development of non-proprietary research, which ispublished in relevant scientific journals and can be accessed by other interestedparties, and propriety research where knowledge and technical data, which can be basis for specific inventions, are not disclosed openly.

Another linkage to scrutinise is the impact of the use of RI on scientific productivity,international recognition and reputation. The availability of advanced equipment hasan effect on shaping how scientific communities organise themselves. It cansignificantly increase the productivity of research teams, as they do not need to seekand arrange experimentation and testing opportunities on limited-availability facilitiesor abroad (e.g. access to Grid computing facilities substitutes the need to queue forsupercomputer service time).


15/22



Open access to RI also induces more regular inward flow of researchers promotingcloser involvement of local teams in international research networks (brain exchange).Such interaction can generate important learning effects for experienced and early-stage researchers alike. Increase in international recognition of leading scientists andthe capacity of research teams can attract further international competitive funding to

the research system. In addition, researcher experience gained during the set-up andoperational phase of RI can have further policy impact through contributions to RIroadmaps.

Figure 9: Indicators for impact on scientific activity


1. Number of articlespublished in the ISI levelinternational scientific journals as a direct resultof research using RI

The aim should be to track the dynamics ofpublishing behaviour due to RI; peer-reviewedarticles applies as an indicator of scientificactivity to most scientific fields; in humanitiesand some sub-fields of social sciences books, book chapters and monographs are more

relevant indicator2. Number ofmethodologies/designsdeveloped

Applies to those scientific fields where thisoutput is relevant to measure scientificproductivity

3. International patentsgranted and publishedpatent applications (alltypes)

Indicator is relevant for most scientific fields innatural sciences, engineering, life sciences andmedicine; in humanities and social sciencesother types of IPR (e.g. copyrights) should beconsidered

4. Number of PhDdissertations completed

Per year of completion and per scientific field;includes those PhD dissertations which have been predominantly or partly-based on the useof RI services or equipment

5. Number of scientific eventsorganised on researchtopics directly relating toRI services

Data on the frequency, types of events,geographic origin and institutional affiliation ofparticipants can provide some proxy on the RIimpact on mutual learning, knowledge exchangeand researcher involvement in major scientificnetworks; some conclusions can be drawn onthe critical mass and synergy effects with otherRI facilities in the region

2.2.5 Broader impacts on society

Regarding wider social impacts, it is important to outline that research infrastructurescan play an important role in scientific communication and scientific education. RIcan be used to inspire school students to learn STEM subjects. For example, large-scale RI may organise open days for the general public or for schools. The policy-decision on investment in RI can also be widely reflected in the press leading toincreased public awareness of science.

Research outcomes from using RI services can lead to concrete innovative products,and services that are taken up and diffused in society (for example, new medicalinstruments, diagnostics, treatments). While such impacts on society are broad,indirect and very difficult to attribute and quantify, it is nevertheless important toscope the potential contribution of RI to solving various societal problems.

There can also be concrete benefits that result from improving local infrastructure,urban planning and community services as the RI investment may revitalise certainareas with important indirect societal benefits. These developments can further shapelocal cultural activities and citizen lifestyles.


16/22



Figure 10: Indicators for impacts on society


1. Number of organised RI opendays for wider public and any

available data on participantsatisfaction with the events

Per year and per target group (e.g.schools/general public); aim to provide an

indication on the impact of publicawareness of science

2. Number of press articles on theinvestment in researchinfrastructure

Per year and per type of publication (printmedia/online articles/specialistpublications)

3. Number of new or improvedproducts, services, solutions as aresult of research using RI thatare diffused in society

This indicator involves especiallysignificant time-lag; the aim to scope thepossible contribution of RI to specificsocietal outcomes

4. Account of improved localinfrastructure, communityservices, increase in local

cultural/recreational activitiesdue to RI

Aim to scope the possible contribution ofRI to the outcomes

3.

Impact assessment methods and tools

3.1 Impacts on economy

To measure the direct economic effect of the RI design and construction phaserequires an analysis of all suppliers that have won the procurement tenders to scopethe contract values per sector of activity and service provider affiliations(local/foreign/multinational). Relevant supply chains should be mapped to determinethe scope of the output, value added and additional employment that has been created

locally and the benefits that have been diffused internationally. Desk analysis of theproposal and contractual documentation and survey among suppliers can be used tocollect this information.

The operational phase requires a comprehensive look at the activities undertaken bythe managers responsible for establishing and running the research facility. Thisinvolves a detailed account of facilities expenditure on salaries, purchase of goods andservices from financial records and direct output generated. Desk research andinterviews with RI managers/coordinators can be used to collate and analyse financialand activity data.

The multiplier effect on the economy is addressed through an input-output analysis.This ‘ripple effect’ includes the indirect economic effects that occur from all secondaryindustries supporting the procured activities and induced effect representing the

changes in consumption of respective households. The input-output model is aquantitative econometric technique that represents the interdependencies betweendifferent sectors of a national/ regional economy. An input-output analysis estimatesthe value of the purchases that flow between the identified supply chains in a givenperiod of time. Building IO model it is possible to derive economic multipliers.

Methodological approaches have been developed on applying cost-benefit analysis toRI projects17, yet these are suited more for ex-ante project evaluation as it takes a longtime period for the effects of RI investment costs to materialise. Nonetheless, theapplicability of CBA type approaches can be explored. For instance, the effects on RI

17 See e.g. Clarke, S. et.al. (2013) Project Preparation and CBA of RDI Infrastructure Projects. JASPERSStaff Working Paper


17/22



users can be scoped through application of contingent valuation techniques.Contingent valuation is a method that uses surveys to draw out the willingness ofactual and potential users to pay for certain services. The method involves assigning ofmonetary values to non-market goods and services based on preferences.

Data requirements

• Overview of contractual information with RI suppliers (design, construction,maintenance, operations, renewal)

• Data on the affiliation of suppliers (local/foreign/multinational)

• Data on employment, wages

• Facilities annual expenditure broken down by function in time series

• Strategic plans and annual activity reports

3.2 Impacts on human resource capacity

To account for the impacts on human resource capacity it is necessary to capture thestructural changes in organisational behaviour that arise as a result of the researchinfrastructure. Monitoring statistics on the number of staff and students attracted, in

particular from abroad, can shed a light on the dynamics of knowledge exchange thatRI is facilitating. It is necessary to understand what are the formal organisationalaspects, how training on RI is carried out and what are the practices for informalknow-how transfer (interaction with technicians, networks). Surveys among students, vising and permanent researchers, technicians and RI managers can be employed fornecessary data gathering. Particularly important is to explore the spaces of studentand researcher interaction with industry. Longitudinal analysis of graduate careerpaths is a valuable approach.

In addition to the available monitoring statistics and information from surveys, datacollection strategies for tracing impacts on human resource capacity should involveextensive qualitative research techniques. These can include semi-structuredinterviews with facility owners and a representative population of researchers andstudents. A case study approach is an appropriate way to capture the dominantpatterns of knowledge diffusion and the degree of social capital formation that theestablishment of RI has promoted.

Data requirements

• Relevant monitoring data, including e.g. statistics on the numbers of attracted staff andstudents, (locally and from abroad) and career paths of graduates that have used RI fortraining or MA/PhD thesis

• Database of concluded service contracts

• Documentation of human resource development strategies, user guides and otherdocumented evidence

• Annual activity reports

3.3

Impacts on innovationImpacts on innovations arise via multiple impact pathways. The most feasible methodis to apply quantitative analysis to scope economic effects of innovation (e.g. value ofIP portfolio) and use qualitative investigation to understand in detail the knowledgespillovers. Monitoring data on knowledge transfer activities can be supplemented withsupplier surveys (for tracing procurement-led innovation and skills formation) and in-depth interviews (with research teams, spin-offs, relevant industry representatives).

Narrative descriptions of individual case studies that look in detail into specific aspectsof innovation impact (e.g. impact via prototyping, industrial demonstrators) would besupportive for the overall assessment. If any routine user surveys are executed by RImanagement this would be very useful source of material to better understand userexperiences, motivations and wider effects of RI use.


18/22



Data requirements

• Relevant official statistics and structured monitoring data (i.e. regular progress reportsand data for measure project level key performance indicators)

•

Database of users (academic, industrial) over the lifetime of the facility• Database of supplier contracts (held either by the funding agency or by beneficiary

higher education or public research institutes)

3.4 Impacts on scientific activity

Impacts on scientific activity can be analysed through a bibliometric analysis of theresearch teams working on the RI. Especially valuable would be to see the evolution ofco-publications over a period of time. A time-series/network analysis of theparticipation in international research project consortia (FP7, H2020) may shed somelight on the visibility of research groups on the international research landscape.

All quantitative techniques should be supplemented with in-depth interviews with keyresearchers to deepen, test and validate the claims. There are many limitations, as the

knowledge spillovers that arise as a result of the intellectual advances may be largelyinvisible to facility owners and stakeholders.

Data requirements

• Database of researchers working on the RI

• Relevant official statistics and structured monitoring data

• Annual activity reports

3.5 Impacts on society

The broader impacts that research infrastructure investment have on society is the

least scrutinised area due to the fact that such impacts are difficult to trace andquantify. While direct causal chains are almost impossible to establish, social sciencemethods can be employed to estimate the potential contribution of RI. Surveys amongschools, public sector, relevant stakeholder groups can be used for this purpose.Further interviews with relevant target groups for RI communication and PR activitiesand the involved researchers can be undertaken. Impact on STEM education andpublic understanding of science can be best explained through analytical narrative inthe form of case studies.

Data requirements

• Documented materials on RI public relations activities (open days, public lectures,seminars)

• Available data from satisfaction and feedback surveys of participants to RIcommunication events

• Track of media publications/online content regarding RI


19/22



Figure 11: Implementation of an impact assessment of research infrastructure


20/22



3.6 IA implementation process, its challenges and limitations

Figure 11 provides an overview the IA implementation steps and use of analyticaltechniques. During the inception phase desk research is performed on all availabledocumented evidence to determine the information gaps that should be covered

through additional data collection methods. In the data collection phase surveys arelaunched beforehand. In-depth interviews and the application of quantitativetechniques can be run in parallel. Case studies are usually chosen after the firstpreliminary findings to ensure that the most relevant aspects and impact areas arescrutinised in detail. The synthesis phase of findings ideally should involve a peerreview and an international expert team to ensure appropriate benchmarking of RIperformance. A full impact assessment takes usually around six to nine month period.

Data shortages are likely to be the most important challenge due to the limited datacaption routines during the lifetime of research facility. Database development withappropriate quantitative data is a foundation for carrying out an in-depth impactassessment. This calls for anticipatory collection of relevant data at an early stage of RIproject implementation. The principle should be that the amount if work forresearchers is kept to a minimum. RI management should be encouraged to regularlyassess key sources of information and collect relevant data on an on-going basis. Theseefforts should include as a minimum data which is readily available or which can becollected automatically, e.g. CVs, completed doctorates, concluded contracts, etc.Bearing in mind the challenge of data availablity and quality, the expectations have to be realistic in terms of the quantifiable findings that can result from the evaluation.

Further challenges and limiting factors to IA of RI include:

• The difficulty of capturing multidimensional and complex research outcomes andimpacts with simple metrics that are used as core indicators. For instance, theaccuracy of input-output models to determine multipliers is questioned. Findingsfrom the application of quantitative methods should be well-balanced withqualitative insights and domain expert validation.

•

Issue of displacement – or the extent to which economic benefits occur at theexpense of other economic activities – is hard to address in the analysis.

• Delineation between short-term, medium and long-term effects will be verydifficult due to restraints in data availability in time series.

• It is hard to separate between RI induced benefits and those fostered by otherdeterminants. The impacts of RI should be accompanied by analysis of parallelinitiatives in higher education and support measures for enterprises. Nationallegal frameworks (e.g. rules for skilled migration, IPR) are all importantdeterminants.

The establishment of systematic evaluation practice for investment in researchinfrastructures is key to expand reference material and foster learning for morecomprehensive analyses of RI impacts.


21/22



Bibliography

Autio, E. (2014) Innovation from Big Science: Enhancing Big Science Impact Agenda.Report to UK Department for Business, Innovation & Skills. Available at:https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/288481/bis-14-618-innovation-from-big-science-enhancing-big-science-impact-agenda.pdf

Autio, E., Streit-Bianchi, M., and Hameri, A.-P. (2003) Technology Transfer andTechnological Learning Through CERN’s Procurement Activity, CERN ScientificInformation Service. Geneva

Bianchi-Streit, M. et al (1984) Economic utility resulting from CERN contracts (secondstudy). Report of the European Organisation for Nuclear Research (CERN). Availableat: http://indico.cern.ch/event/66952/contribution/0/2/material/paper/0.pdf

Bozeman, B. (2000) Technology transfer and public policy: a review of research and

theory. Research Policy, 29(4–5)

Clarke, S. et.al. (2013) Project Preparation and CBA of RDI Infrastructure Projects.JASPERS Staff Working Paper. Available at:http://www.jaspersnetwork.org/display/for/Project+Preparation+and+CBA+of+RDI+Infrastructure+Projects

European Strategy Forum on Research Infrastructures (2011) Evaluation Report 2011. Available at:http://ec.europa.eu/research/infrastructures/pdf/esfri_evaluation_report_2011.pdf

Fotakis, C. (2010) Analyses of FP7 supported Research Infrastructures initatives in thecontext of ERA. Available at:http://ec.europa.eu/research/evaluations/pdf/archive/fp7-evidence-

base/experts_analysis/c.%20fotakis_-_research_infrastructure.pdfHallonsten, O., Benner, M. and Holmberg. G (2004) Impacts of Large-Scale ResearchFacilities- A Socio-Economic Analysis. A study done at the Research Policy Institute,Lund University. Available at:http://rifi.gateway.bg/upload/docs/public_doc_REPORT_impact_of_large_scale_R I.pdf

von Hippel, E. (1976) The Dominant Role of Users in the Scientific InstrumentInnovation Process, Research Policy 5 (3)

Hickling Arthurs Low Corporation (2013) Return on Investment in Large ScaleResearch Infrastructure. Report for National Research Council Canada. Available at:http://www.triumf.ca/sites/default/files/HAL-ReturnOnInvestmentStudy-May-2013.pdf

Horlings, E. et.al (2012) The societal footprint of big science. Report of the RathenauInstituut. Available at: http://www.rathenau.nl/en/publications/publication/the-societal-footprint-of-big-science.html

National Research Council Canada (2013) Return on Investment in Large ResearchInfrastructure. Available at: http://www.triumf.ca/sites/default/files/HAL-ReturnOnInvestmentStudy-May-2013.pdf

OECD (2014) International Distributed Research Infrastructures: Issues and Options. Available at: http://www.oecd.org/sti/sci-tech/international-distributed-research-infrastructures.pdf

OECD (2014) Report on the Impacts of Large Research Infrastructure on EconomicInnovation and on Society: Case Studies at CERN. Available at:

http://www.oecd.org/sti/sci-tech/CERN-case-studies.pdf


22/22


Rizutto, C. and Wood, J. (eds.) (2013) RAMIRI Handbook. Deliverable of FP7 projectRealising and Managing International Research Infrastructures. Available at:http://www.ramiri-blog.eu/index.php?n=Main.HomePage

Rizzuto, C. (2012) Benefits of Research Infrastructures beyond Science, presentationat ERF Workshop “The Socio-Economic Relevance of Research Infrastructures”, 31May-1 June 2012, Hambourg

Sallee, C.M., Watkins, S.D and Rosaen, A.L. (2011) The Economic Impact of FermiNational Accelerator Laboratory. Anderson Economic Group report to the Universityof Chicago. Available at:https://ovprnl.uchicago.edu/sites/research.uchicago.edu/files/Fermilab_Economic_Impact_Full_Study.pdf

Simmonds, P. et.al (2013) Big Science and Innovation. Technopolis Group report forthe UK Department for Business, Innovation & Skills. Available at:https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/249715/bis-13-861-big-science-and-innovation.pdf

Wissenschaftsrat (2011) Recommendations on Research Infrastructures in

Humanities and Social Sciences. Available at:http://www.wissenschaftsrat.de/download/archiv/10465-11_engl.pdf

Wissenschaftsrat (2013) Report on the Science-Driven Evaluation of Large ResearchInfrastructure Projects for the National Roadmap (Pilot Phase). Available at:http://www.wissenschaftsrat.de/download/archiv/2841-13_engl.pdf

Zuijdam, F. et. al (2011) The role and added value of large-scale research facilities.Technopolis Group report. Available at: http://www.technopolis-group.com/wp-content/uploads/2011/02/1379_Report_Large-scale_Research_Facilities_EN1.pdf

Please cite this publication as:

Griniece E., A Reid and J. Angelis (2015): Guide to Evaluating and Monitoring Socio-Economic Impact of Investment in Research Infrastructures. Technopolis Group.Tallinn. Estonia

Front cover photo: Centre for New Pharmaceutical and Health Technologies of theSantaka Valley, Kaunas, Lithuania. Source: http://www.mitnija.lt/

2015 technopolis group guide to impact assessment of research infrastructures

Documents