overview of this presentation
DESCRIPTION
Experiences with a bibliometric indicator for performance-based funding of research institutions in Norway Gunnar Sivertsen Nordic Institute for Studies in Innovation, Research and Education, Oslo, Norway. Overview of this presentation. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/1.jpg)
Experiences with a bibliometric indicator for performance-based funding of research institutions in Norway
Gunnar Sivertsen
Nordic Institute for Studies in Innovation, Research and Education, Oslo, Norway
![Page 2: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/2.jpg)
Overview of this presentation
1. Performance based funding: Expert panels or indicators or both?
2. Bibliometric data sources
3. A bibliometric indicator covering all fields of research
4. Experiences: Effects, Discussions, Criticisms
5. Results from and questions for two evaluations of the indicator
![Page 3: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/3.jpg)
Overview of this presentation
1. Performance based funding: Expert panels or indicators or both?
2. Bibliometric data sources
3. A bibliometric indicator covering all fields of research
4. Experiences: Effects, Discussions, Criticisms
5. Results from and questions for two evaluations of the indicator
![Page 4: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/4.jpg)
”The dual funding system” (A simplified model excluding international sources, etc.)
![Page 5: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/5.jpg)
Same aims, methods, and effects?
![Page 6: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/6.jpg)
Panel evaluation and bibliometric methods
![Page 7: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/7.jpg)
Used for (indirect) project and programme funding via research councils
Used for systematic (repeated) research evaluations of large programmes, disciplines and/or institutions
Panel evaluation and bibliometric methodsin the Scandinavian countries
Used as information by panel evaluations
Used for direct performance-based funding of research institutions
![Page 8: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/8.jpg)
Used for (indirect) project and programme funding via research councils
Used for systematic (repeated) research evaluations of large programmes, disciplines and/or institutions
Panel evaluation and bibliometric methodsin the Scandinavian countries
Used as information by panel evaluations
Used for direct performance-based funding of research institutions
![Page 9: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/9.jpg)
Motivation in Norway: The need for more emphasis on research in the overall funding model for the Higher Education Sector
Budget for HEInstitutions
Basic: 60% Education: 25% Research: 15%
Strategic Performance based
Researchers
Dissertations
External funding
“Study points”since 2002
?
![Page 10: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/10.jpg)
Motivation in Norway: The need for more emphasis on research in the overall funding model for the Higher Education Sector
Budget for HEInstitutions
Basic: 60% Education: 25% Research: 15%
Strategic Performance based
Scientific publications
Dissertations
External funding
Since 2006:“Publication points”
“Study points”since 2002
![Page 11: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/11.jpg)
Motivation in Norway: The need for more emphasis on research in the overall funding model for the Higher Education Sector
Budget for HEInstitutions
Basic: 60% Education: 25% Research: 15%
Strategic Performance based
Scientificpublications
Dissertations
External funding
Since 2006:“Publication points”:
Less than 2 %of total budget
“Study points”since 2002
![Page 12: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/12.jpg)
Panel evaluation and bibliometric methods
![Page 13: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/13.jpg)
1986
![Page 14: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/14.jpg)
1986
2003
![Page 15: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/15.jpg)
2003
2005
1986
![Page 16: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/16.jpg)
2003
2005
1986
2009
![Page 17: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/17.jpg)
2003
2005
1986
2009
2009
![Page 18: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/18.jpg)
2009
2005
1986
2009
2009
![Page 19: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/19.jpg)
2009
2004
1986
2009
2009
2015
![Page 20: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/20.jpg)
2009
2004
1986
?
2008
2015
![Page 21: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/21.jpg)
Overview of this presentation
1. Performance based funding: Expert panels or indicators or both?
2. Bibliometric data sources
3. A bibliometric indicator covering all fields of research
4. Experiences: Effects, Discussions, Criticisms
5. Results from and questions for two evaluations of the indicator
![Page 22: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/22.jpg)
Indexed journals
Journals and series
Books
![Page 23: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/23.jpg)
Österreichische Zeitschrift für Politikwissenschaftis covered …
![Page 24: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/24.jpg)
But Österreichische Zeitschrift für Soziologie is not covered
![Page 25: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/25.jpg)
Swedish Journal of Sociology is covered …
![Page 26: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/26.jpg)
But Swedish Journal of Political Science is not covered
![Page 27: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/27.jpg)
Principles behind the use of institutional data in a national information system
•Completeness: All scholarly publications should be included
•Transparency: Every institution can see and check all other institutions’ data. The national database is also online and open to society at large.
•Participation: The indicator is developed and maintained in collaboration between the institutions and the authorities
•Multiple use of the data: CV’s, applications, evaluations, annual reports, internal administration, bibliography for Open Archives, links to full text, etc.
![Page 28: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/28.jpg)
![Page 29: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/29.jpg)
References imported from ISI
1351-510
![Page 30: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/30.jpg)
References imported from ISI
References to book and article in book added
![Page 31: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/31.jpg)
Overview of this presentation
1. Performance based funding: Expert panels or indicators or both?
2. Bibliometric data sources
3. A bibliometric indicator covering all fields of research
4. Experiences: Effects, Discussions, Criticisms
5. Results from and questions for two evaluations of the indicator
![Page 32: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/32.jpg)
Delimitation of publication data by definition
Definition
A scientific or scholarly publication must:
1. present new insight
2. in a form that allows the research findings to be verified and/or used in new research activity
3. in a language and with a distribution that makes the publication accessible for a relevant audience
4. in a publication channel with peer review*)
*) In addition: Publication channels with authors coming mainly from only one institution are not included.
![Page 33: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/33.jpg)
Three main publication types
Channel
Type
E.g. scientific article in Nature or scholarly ISBN-title on Oxford University Press.
1) Article in ISSN-title2) Article in ISBN-title
3) ISBN-title
![Page 34: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/34.jpg)
Dynamic records of scientific and scholarly ISSN-titles and ISBN-publishers
0001-9887 Africa Today
0001-9909 African Affairs
1062-4783 African American Review
0263-0338 African Archaeological Review
0001-9933 African Arts
1017-6772 African Development Review
0145-2258 African Economic History
1021-3589 African Entomology
1472-5843 African Identities
1684-5315 African Journal of Biotechnology
1021-9730 African Journal of Crop Science
0141-6707 African Journal of Ecology
1438-7890African Journal of Environmental Assessment and Management
1684-5378African Journal of Food, Agriculture, Nutrition
1015-8618 African Journal of Neurological Sciences
0065-4000 African Literature Today
CABI Publishing
Cambridge University Press
Cappelen Akademisk Forlag
Carfax Publishing
Carl Heymanns Verlag
Carlsson bokförlag
Catholic University of America Press
Central European University Press
Channel View Publications
Chinese University Press
Christian Ejlers´s Forlag
22,000 ISSN-titles and 1,200 book publishers so far
![Page 35: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/35.jpg)
All channels can be searched, and new channels can be suggested
![Page 36: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/36.jpg)
Publication channels on two levels
Level 2: Higher points to 20 % of the publications
Level 1: Normal points to 80 % of the publications
Level 2 represents the most prestigous international channels
Defined in each discipline by expert panels informed by indicators
![Page 37: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/37.jpg)
Publication points
Publication type Level 1 Level 2
Article in ISSN-title 1 3
Article in ISBN-title 0,7 1
ISBN-title 5 8
Publication points are fractionalized if the publication originates from more than one institution
![Page 38: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/38.jpg)
Overview of this presentation
1. Performance based funding: Expert panels or indicators or both?
2. Bibliometric data sources
3. A bibliometric indicator covering all fields of research
4. Experiences: Effects, Discussions, Criticisms
5. Results from and questions for two evaluations of the indicator
![Page 39: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/39.jpg)
Publication points in Norway’s Higher Education Sector 2004-2011
0
2 000
4 000
6 000
8 000
10 000
12 000
14 000
2004 2005 2006 2007 2008 2009 2010 2011
0 %
5 %
10 %
15 %
20 %
25 %Level 2
Level 1
Level 2 (%)
![Page 40: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/40.jpg)
Shares in the world’s scientific output (ISI 2000-2011)Source: National Science Indicators (NSI), Thomson Reuters
![Page 41: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/41.jpg)
Shares in the world’s scientific output (ISI 2000-2011)Source: National Science Indicators (NSI), Thomson Reuters
![Page 42: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/42.jpg)
Relative citation rates (ISI 1999-2010)Weighted according to research profiles.Source: National Science Indicators (NSI), Thomson Reuters
![Page 43: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/43.jpg)
Effects on the institutional level
The institutions have stronger incentives to facilitate research for their researchers
Research is now perceived as a common and institutional responsibility, not only as an individual task
New publications receive attention, not only from external peers, but also internally from the institution
Research management improves with the aid of complete bibliometric information about the research activities
![Page 44: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/44.jpg)
An example of effect at department level: Poster in the toilets of The Centre of Excellence in linguistics at the University of Tromsø
![Page 45: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/45.jpg)
Criticism A: Use and abuse of the indicator on the local or individual level
![Page 46: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/46.jpg)
Criticism B: Will research published in the Norwegian language disappear?
![Page 47: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/47.jpg)
Trends in use of English and WoS coverage in the humanities and social sciencesPercentages of all scholarly journal articles from Flemish (F) and Norwegian (N) universities that were published in English and covered by Web of Science 2005-2009
![Page 48: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/48.jpg)
Criticism C: Are the best publications getting most points?
![Page 49: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/49.jpg)
Criticism D: What happens to research quality? Should we measure impact?
![Page 50: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/50.jpg)
Overview of this presentation
1. Performance based funding: Expert panels or indicators or both?
2. Bibliometric data sources
3. A bibliometric indicator covering all fields of research
4. Experiences: Effects, Discussions, Criticisms
5. Results from and questions for two evaluations of the indicator
![Page 51: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/51.jpg)
Results from an evaluation in Denmark in 2012
Difficult to measure effects after only three years
Keep it, and keep it simple / add citations
Increase transparency and statistical use
![Page 52: Overview of this presentation](https://reader036.vdocument.in/reader036/viewer/2022062721/568137c2550346895d9f609a/html5/thumbnails/52.jpg)
Questions for an evaluation in Norway in 2013
Does the indicator balance well between different domains of research?
Are the two levels appropriate?
What happens to research collaboration?
Changes in the publication patterns?
How is the indicator perceived by researchers?
The impact on research and on other activities
Local use of the indicator