european trends in the use of evaluations and indicators for performance-based funding of research...

European Trends in the Use of Evaluations and Indicators for Performance-Based Funding of Research Institutions

Gunnar Sivertsen

Nordic Institute for Studies in Innovation, Research and Education, Oslo, Norway

Overview of this presentation

1. The dual funding system in Europe

2. Research evaluation and performance indicators

3. Evaluation-based versus indicator-based institutional funding

4. New trends:

a. Evaluation of societal impact

b. Data from Current Research Information Systems

5. Conclusions

”The dual funding system” (Public sources)

Government

Direct grants

Historical, Political

Strategic

Performance based

Research institutions

Research Councils

Projects and Programmes

Competition

Research Evaluation

The balance?

Government

Direct grants


Strategic

Performance based


Research Councils


Competition

Research Evaluation

The mix?

Government

Direct grants


Strategic

Performance based


Research Councils


Competition

Research Evaluation

Same aims, methods, and effects?

Government

Direct grants


Strategic

Performance based


Research Councils


Competition

Research Evaluation





4. New trends:



5. Conclusions

Research Evaluation and Performance Indicators

Bibliometrics may inform, but not replace peer review

A. Research Evaluation without funding implications


Bibliometrics informs the evaluation. Evaluation results are not translated into funding

B. Research Evaluation with funding implications (PBF)


Bibliometrics informs the evaluation. Evaluation results are translated into funding

C. Performance-based funding (PBF) without Research Evaluation

Bibliometrics is translated into funding. Often combined with other indicators.





4. New trends:



5. Conclusions

Examples of C: Indicator-based models

Bibliometrics is

translated into funding.

Often combined with other indicators.

1990

Examples of B: Evaluation-based models

2010

Bibliometrics informs the evaluation. Evaluation results are translated into funding

1986

2003

2018?

2018?

Examples of B and C: Combinations in practice

2010

Bibliometrics is translated into evaluation. Evaluation results are translated into funding

2003

Examples of A: Evaluation without funding implications

2010

Bibliometrics informs the evaluation. Evaluation results are not translated into funding





4. New trends:



5. Conclusions

Phases in the development of the UK model

1986

The conclusion after an open consultation in UK in 2006: Is it possible to use metrics instead or as a supplement?

Bibliometrics (citation indicators) may inform, but not replace peer review

The conclusion after an open consultation in UK in 2006: Is it possible to use metrics instead or as a supplement?

We are not covered!!

2018?

2014

• Evaluation panels may be provided

with bibliometrics upon request.

• Use of journal impact factors and

other bibliometric indicators not supplied through the REF

administration is forbidden.

2018?

2017?

2014

2003

Metrics may inform, but not replace peer review

Impact template as Word file (6,975 submissions)

Impact template as Word file (6,975 submissions)

“For the impact component of the REF, it is not currently feasible to use quantitative indicators in place of narrative impact case studies, or the impact template.”





4. New trends:



5. Conclusions

Indicator-based models are increasingly based on institutional or national Current Research Information Systems (CRIS)

Principles behind the use of institutional data in a shared national Current Research Information System • Completeness: All scholarly

publications and other results from research are included

• Transparency: Every institution can see and check all other institutions’ data. The national database is also online and open to society at large.

• Multiple use of the data: CV’s, applications, evaluations, annual reports, internal administration, bibliography for Open Archives, links to full text, etc.

Indexed journals

Journals and series

Books

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Health

Sciences

Natural

Sciences

Engineering Social Sciences Humanities

Coverage of 70,500 scholarly publications from the

higher education sector in Norway 2005-2012.

ScopusWeb of Science

Coverage of scholarly publications in Norway’s CRIS (100 per cent) compared to coverage in WoS and Scopus

Both Elsevier and Thomson Reuters are now integrating the citation databases with research management tools and CRIS-systems





4. New trends:



5. Conclusions

Conclusions: European Trends in the Use of Evaluations and Indicators for

Performance-Based Funding of Research Institutions

Both evaluation-based and indicator-based solutions prevail

PBF is effective even if introduced as a small share of the total funding. The discussion is about positive versus negative effects

Difficult to identify “best practice” or overall positive/negative effects

Increasingly important:

Evaluation of societal impact

Use of CRIS systems

european trends in the use of evaluations and indicators for performance-based funding of research...

Presentations & Public Speaking