Informing the peer

review: Bibliometrics

Gunnar Sivertsen - NIFU

www.metodika.reformy-msmt.cz

Informing the peer review: Bibliometrics Gunnar Sivertsen

Nordic Institute for Studies in Innovation, Research and Education, Oslo

1

Overview of this presentation

1. What is bibliometrics?

2. Bibliometric data sources

3. Bibliometrics in performance-based institutional funding

a. Indicator-based models

b. Evaluation-based models

4. Bibliometrics informing research evaluation: Good practice

2

Overview of this presentation

1. What is bibliometrics?

2. Bibliometric data sources

3. Bibliometrics in performance-based institutional funding

a. Indicator-based models

b. Evaluation-based models

4. Bibliometrics informing research evaluation: Good practice

3

Bibliographic data sources are normally

used to search and retrieve the scientific

literature

4

Bibliographic data

sources

Researchers as authors

Publi-

cations

Publi-

cations

Researchers as readers

Bibliometrics is the use of bibliographic

data sources to study the research behind

the scientific literature

5

Bibliographic data

sources

Researchers as authors

Publi-

cations

Publi-

cations

Researchers as readers

Bibliometrics is possible because research

must be published

6

Bibliographic data

sources

Researchers as authors

Publi-

cations

Publi-

cations

Researchers as readers

… and because relevant references must

be given

7

Bibliographic data

sources

Researchers as authors

Publi-

cations

Publi-

cations

Researchers as readers

… and because co-authors must be

credited

8

Bibliographic data

sources

Researchers as authors

Publi-

cations

Publi-

cations

Researchers as readers

The publication

9

The publication: Where published?

10

The publication: When published?

11

The publication: About what?

12

The publication: By whom?

13

The publication: From where? Collaboration?

14

The publication: Citation impact?

15

Research Evaluation and Bibliometrics

16

Bibliometrics may inform, but not replace peer review

Overview of this presentation

1. What is bibliometrics?

2. Bibliometric data sources

3. Bibliometrics in performance-based institutional funding

a. Indicator-based models

b. Evaluation-based models

4. Bibliometrics informing research evaluation: Good practice

17

18

Indexed

journals

Journals

and series

Books

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Health Sciences Natural Sciences Engineering Social Sciences Humanities

Coverage of 70,500 scholarly publications from the higher education sector in Norway 2005-2012.

Scopus Web of Science

Incomplete coverage of international

journals.

Very limited coverage of books.

Random or no coverage of the national

level (books and journals)

Österreichische Zeitschrift für

Politikwissenschaft

is covered …

19

But Österreichische Zeitschrift für

Soziologie is not covered

20

Swedish Journal of Sociology

is covered …

21

But Swedish Journal of Political Science is

not covered

22

Specialized bibliographic data sources may have broader coverage, but do not include citations and

address information

23

Google Scholar is of great help to the scientist as reader but not to the bibliometrician: No control over data and methods

24

• Jedna

• Dvě

• Tři

• čtyři

25

Overview of this presentation

1. What is bibliometrics?

2. Bibliometric data sources

3. Bibliometrics in performance-based institutional funding

a. Indicator-based models

b. Evaluation-based models

4. Bibliometrics informing research

26

a. Performance-based funding (PBF)

without Research Evaluation

27

Bibliometrics is translated into

funding.

Most often combined with other

indicators.

b. Research Evaluation with funding

implications (PBF)

28

Bibliometrics may inform, but not replace peer review

Bibliometrics informs the evaluation.

Evaluation results are translated into

funding

c. Research Evaluation without funding

implications

• Jedna

• Dvě

• Tři

• čtyři

29

Bibliometrics may inform, but not replace peer review

Bibliometrics informs the evaluation.

Evaluation results are not translated

into funding

a) Examples of indicator-based models

b) Examples of evaluation-based models

31

c) Examples of evaluation-based models

without PBF

32

b) Data sources for evaluation-based models:

Combination of CRIS and WoS/Scopus

33

Overview of this presentation

1. What is bibliometrics?

2. Bibliometric data sources

3. Bibliometrics in performance-based institutional funding

a. Indicator-based models

b. Evaluation-based models

4. Bibliometrics informing research evaluation: Good practice

34

35

2018?

2014

• Evaluation panels may be provided with

bibliometrics upon request.

• It is forbidden for the panels to use

Journal Impact Factors or other

bibliometric indicators not supplied

through the REF administration.

Peer Review accompanied by bibliometrics

allows for appropriate treatment of different

levels of assessment

36

▼ global developments

▼ national R&D systems

▼ policies

▼ cross-sectional fields

▼ research and grant programs

▼ academic fields

▼ universities, research institutes, funding agencies

▼ university institutes/departments

▼ target/status groups

▼ research groups

▼ individuals

Macro

Micro

Meso

Peer Review and bibliometrics have

different strengths and limitations; they

supplement each other

37

▼ global developments

▼ national R&D systems

▼ policies

▼ cross-sectional fields

▼ research and grant programs

▼ academic fields

▼ universities, research institutes, funding agencies

▼ university institutes/departments

▼ target/status groups

▼ research groups

▼ individuals

Macro

Micro

Meso

Peer review

Bibliometrics

Peer Review allows for collecting and

relating more information

38

Input

Output

Efficiency

Human resources

Financial resources

Infrastructure (equipment, laboratory space etc.)

Prizes Bibliometric

indicators

Patent

indicators

Third party

funding

Performance Structure

Activity Reception Cognitive Collaboration

Authorship Citation

Various Input / Output

relations Inter and intra

institutional

comparisons

b. Research Evaluation with funding

implications (PBF)

39

Bibliometrics may inform, but not replace peer review

Bibliometrics informs the evaluation.

Evaluation results are translated into

funding

The “Leiden Manifesto” (Draft September 2014)

1. Metrics properly used to support assessments; they do not substitute for judgment. Everyone

retains responsibility for their assessments.

2. It is easy to underestimate the difficulty of constructing accurate data. Spend the time and

money required to produce data of high quality. Those mandating use of metrics should be able

to provide assurance that the data is accurate.

3. Metrics should be transparent, the construction of the data should follow a clearly stated set of

rules. Everyone should have access to the data.

4. Data should be verified by those evaluated, who should be offered the opportunity to

contribute explanatory notes if they wish.

5. Sensitivity to field differences is important. Metrics will differ by field. Humanists will not be

able to use citation counts; computer scientists will need to ensure conference papers are

included; and chemists will look the best in raw metrics constructed from Web of Science data.

The state-of-the-art is to select a suite of possible indicators and allow fields to choose among

them.

6. Normalize data to account for variation in citation and publication rates by field and over time.

7. Metrics should align with strategic goals.

40

Děkujeme za pozornost!

www.metodika.reformy-msmt.cz