An overview of main bibliometric indicators:

Dag W. Aksnes

Data sources, methods and applications

Nordic Institute for Studies in Innovation, Research and Education

Common indicators

Number of publications used as an indirect measure of knowledge production

The extent to which the publications have been cited in the subsequent scientific literature used as an indicator scientific impact and international visibility

Co-authorship used as an indicator of collaboration, e.g. the extent of international collaboration

3

Common indicators in bibliometric studies of research

Indicator Data

Research output Publications per unit

Research profile Publications per field

Scientific impact Number of citations

Research collaboration Co-authorship

4

Common data sources for bibliometric indicators

Web of Science (traditionally)

Scopus (increasingly)

Various field specific databases (less often, field studies)

Google scholar (increasingly, but controversial)

CRIS-databases (new options)

5

Database requirements for bibliometric indicators (simplified)

Requirements WoS Scopus Google Scholar

CRIStin CRIS (prior)

Adapted for bibliometric studies + + - 0 -

Complete indexing of all authors and author addresses

+ + - 0, + -

Citation counts + + + - -

Transparent and justified selection procedure of source material

0,+ 0,+ - + -

Coverage of natural sciences and medicine

+ + + + 0

Coverage of engineering 0 0 + + 0

Coverage of A&H and SS - - + + 0

Coverage of books and monographs

- - + + 0

CRIS-data Norway

Data on all types of scientific/scholarly publications in all fields of research are collected trough a documentation system (CRIStin).

In the database the publication output is reported by the institutions as ordinary bibliographic references.

A dynamic authority record of controlled scientific and scholarly publication channels ensures that references to non-scientific publications are not entered into the system.

Publication data from professional bibliographic data sources (e.g. Web of Science) are imported to the documentation system in order to facilitate the registration of publications by the employees.

7

Database requirements for bibliometric indicators (simplified)

Requirements WoS Scopus Google Scholar

CRIStin CRIS (prior)

Adapted for bibliometric studies + + - 0 -

Complete indexing of all authors and author addresses

+ + - 0, + -

Citation counts + + + - -

Transparent and justified selection procedure of source material

0,+ 0,+ - + -

Coverage of natural sciences and medicine

+ + + + 0

Coverage of engineering 0 0 + + 0

Coverage of A&H and SS - - + + 0

Coverage of books and monographs

- - + + 0

9

Database requirements for bibliometric indicators (simplified)

Requirements WoS Scopus Google Scholar

CRIStin CRIS (prior)

Adapted for bibliometric studies + + - 0 -

Complete indexing of all authors and author addresses

+ + - 0, + -

Citation counts + + + - -

Transparent and justified selection procedure of source material

0,+ 0,+ - + -

Coverage of natural sciences and medicine

+ + + + 0

Coverage of engineering 0 0 + + 0

Coverage of A&H and SS - - + + 0

Coverage of books and monographs

- - + + 0

Publication types* in five major areas, Norwegian universities

Natural Sci

Health Sci

Engineering

Social Sci

Humanities

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

ISI articles Non-ISI articles Articles in books Books

*) Weighted: 1 book = 5 articles (N=24,154)

11

Coverage of 50 000 journal articles by Norwegian authors*

Health Sciences Natural Sciences Engineering Social Sciences Humanities0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Scopus Web of Science

*) Accredited scientific and scholarly journals, CRIStin-data 2005-2012, Source: Gunnar Sivertsen (preliminary analysis)

Proportion of WoS indexed publications, selected SSH-departments, University of Bergen

Dep of Biological and Medical Psychology

Dep of Economics

Dep of Education and Health

Dep of Psychosocial Science

Dep of Geography

Dep of Clinical Psychology

Dep of Social Anthropology

Dep of Sociology

Dep of Information Science and Media Studies

Dep of Archaeology, History, Culture studies and Religion

Dep of Administration and Organization Theory

Dep of Comparative Politics

Dep of Philosophy

Dep of Foreign Languages

Dep of Linguistic, Literary and Aesthetic studies

0% 10% 20% 30% 40% 50% 60% 70% 80%

13

Common indicators in bibliometric studies of research

Indicator Data

Research output Publications per unit

Research profile Publications per field

Scientific impact Number of citations

Research collaboration Co-authorship

Addresses are the basis for indicators of:

-Publication output per country or research unit

15

Total publication output 2012 (WoS)

USAChin

a

Great

Brit

ain

Germ

any

Japa

n

Franc

e

Canad

a

South

Kor

ea

Austra

lia

Nethe

rland

s

Switzer

land

Sweden

Belgium

Denm

ark

Austri

a

Norway

Finlan

d

Irelan

d0

50000

100000

150000

200000

250000

300000

350000

400000

National publication profile biology (Norway)

University of Bergen9%

University of Oslo9%

University of Tromsø10%

UMB13%

NTNU9%

Other HE-institutions10%

Industry5%

IMR7%

Nofima6%

NINA5%

Institute sector - other insti-tutes14%

Hospitals2%

19

Counting of publications – methodological issues

The question of how to handle papers involving co-authorship in general and international co-authorship in particular has been a recurrent issue in the literature on the use of bibliometric indicators

Whole counting and fractional counting have been considered complementary methods

According to Moed (2005), the integer or whole count method can be interpreted to measure participation, while the fractional counting method measures the number of papers creditable to a country.

20

Counting of publications – methodological issues

Introduction of additional weighting systems, e.g. of first-authorship

Weighting of books

Publication types to be included

Publication channels - two levels

Level 2- is giving extra weight includes only the leading and most selective international journals and publishers - 20 % of the total

Level 1 80% of publications -”normal” wheight

In the Norwegian system journals and publishers are divided into two levels

Funding model Norway: Weigted publication points based on publication level and publication channel:

Level 1 Level 2

– Articles in scientific/scholarly journals: 1 3 – Articles in scientific/scholarly books: 0.7 1 – Monographs: 5 8

23

Common indicators in bibliometric studies of research

Indicator Data

Research output Publications per unit

Research profile Publications per field

Scientific impact Number of citations

Research collaboration Co-authorship

Journal titles are the basis for indicators of:

- Allocation of publications to fields of research- Research profiles- Field normalization

25

Number of articles* by field, 2005-2009

Ag

ricu

lture

Bio

log

y

Bio

me

dic

ine

Ch

em

istr

y

En

gin

ee

rin

g &

M

ate

ria

ls S

...

Ge

osc

ien

ces

He

alth

Sci

e...

Ph

ysic

s &

M

ath

em

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

1400

1500

1600

1700

University of Bergen

University of Oslo

*) Indexed in Web of Science

26

Relative Specialization Index for ‘traditional’ universities (blue line equals the world average) (2005-09)

UiB UiO

NTNU UMB

27

Common indicators in bibliometric studies of research

Indicator Data

Research output Publications per unit

Research profile Publications per field

Scientific impact Number of citations

Research collaboration Co-authorship

Citations are the basis for indicators of:

- Impact (field normalized)

Citation rates by field (2005, 5 year-window)

Discipline Cit/pub Uncited Discipline Cit/pub UncitedMolecular Biology & Genetics 22.4 4 %Geosciences 8.8 12 %

Immunology 20.7 3 %Physics 8.4 17 %

Microbiology 15.6 4 %Plant & Animal Science 6.9 16 %

Biology & Biochemistry 15.1 7 %Agricultural Sciences 6.9 16 %

Space Science 14.2 13 %Economics-Business 5.6 20 %

Clinical Medicine 12.5 10 %Social Sciences, general 5.0 22 %

Chemistry 10.7 13 %Engineering 4.6 26 %

Environment/Ecology 10.3 9 %Mathematics 3.3 29 %

Psychiatry/Psychology 10.2 11 %Computer Science 2.9 48 %

Number of citations per year per article

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 250

0.5

1

1.5

2

2.5

3

3.5

4

Agriculture Economics

Molecular biology

Number of years after publication

Av

g n

um

be

r o

f c

ita

tio

ns

pe

r a

rtic

lel

Cumulative share of citations and publications, % (2005 - 5 years cit. win)

0 10 20 30 40 50 60 70 80 90 1000

10

20

30

40

50

60

70

80

90

100

Share of journal articles

Sh

are

of

cit

ati

on

s

Citations counts – different sources

ARTL_TITLE JRNLFULL VOL YEARCitations - Web

of Science Citations - ScopusCitations - Google

scholarA macro study of self-citation SCIENTOMETRICS 56 2003 90 102 170

Characteristics of highly cited papersRESEARCH EVALUATION 12 2003 57 62 95

Peer reviews and bibliometric indicators: a comparative study at a Norwegian university

RESEARCH EVALUATION 13 2004 49 54 109

Does self-citation pay? SCIENTOMETRICS 72 2007 41 48 66

Scientific productivity and group size: A bibliometric analysis of Norwegian microbiological research SCIENTOMETRICS 49 2000 32 38 57

The effect of highly cited papers on national citation indicators SCIENTOMETRICS 59 2004 30 32 58

Citation rates and perceptions of scientific contribution JASIST 57 2006 29 37 74

Validation of bibliometric indicators in the field of microbiology: A Norwegian case study SCIENTOMETRICS 49 2000 16 13 18Researchers' perceptions of citations RESEARCH POLICY 38 2009 15 17 21

When different persons have an identical author name. How frequent are homonyms? JASIST 59 2008 9 12 14

The Structure and Development of Polar Research (1981-2007): a Publication-Based Approach

ARCTIC ANTARCTIC AND ALPINE RESEARCH 41 2009 4 4 8

Ranking national research systems by citation indicators. A comparative analysis using whole and fractionalised counting methods

JOURNAL OF INFORMETRICS 6 2012 4 6 10

Are Female Researchers Less Cited? A Large-Scale Study of Norwegian Scientists

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY 62 2011 2 3 9

Science policy and the driving forces behind the internationalisation of science: The case of Norway

Science and Public Policy 35 2008 4 8

Total 378 432 717

Google scholar vs Web of science

0 10 20 30 40 50 60 70 80 90 1000

20

40

60

80

100

120

140

160

180

f(x) = 1.8777801425987 xR² = 0.982384912054357

Web of Science

Go

og

le s

ch

ola

r

Relative citation index

Relative citation index field = (Observed Citation Rate/World Citation Average of Subfield)X100

Relative citation index journal = (Observed Citation Rate/Average citation rate of journal)X100

Example (shown for one article only):

– A regular journal article in Acta Chrystallographica E, 2005 has been cited 3 times.

– The articles published in Acta Chrystallographica E were in contrast cited 1.74 times on average this year.

– The Relative citation index – journal is: (3/1.74)*100 = 172. – The world-average citation rate for the subfield which this journal is

assigned to is 3.7 for articles published this year. The Relative citation index – field is: (3/3.7)*100 = 81

36

Field normalized relative citation indexes, 2008-2011

Switzerland

Denmark

Netherlands

Great Britain

USA

Sweden

Belgium

Norway

Canada

Ireland

Finland

Australia

Austria

Germany

France

China

South Korea

Japan

50 60 70 80 90 100 110 120 130 140 150 160

37

Field normalized relative citation indexes of universities and university hospitals (2005-2008) across subject fields

Norway 1.22 1.17 0.96 0.93 1.05 1.12 1.09 1.23 1.08

Norwegian University of Science and Technology

1.16 1.05 1.13 0.87 1.02 0.51 1.03 1.42 1.07

University of Bergen 1.11 1.11 0.94 0.97 1.27 1.38 1.08 1.03 1.11

University of Oslo 1.14 1.32 1.00 0.95 1.07 1.12 1.12 1.20 1.11

University of Stavanger 0.69 0.75

University of Tromsø 1.54 1.28 0.79 1.16 1.01 0.94 0.92 1.04

St.Olavs Hospital 1.08 1.07

University Hospital North Norway 1.05 1.02

University of Bergen Hospitals 0.67 1.12 1.04

University of Oslo Hospitals 1.02 1.13 1.11

University

Agri

culture

, Fis

heri

es &

Fore

stry

Bio

logy

Bio

medic

ine

Chem

istr

y

Engin

eeri

ng &

Mate

rials

Sci

ence

s

Geosc

ience

s

Hea

lth S

cien

ces

Physi

cs &

Math

em

atics

Tota

l

H-index

Articles ranked by decreasing citation numbers. A scientist with an index of h has published h papers each of which has been cited by at least h times – can be manually determined using citation databases or using

automatic tools– attempts to measure both the scientific productivity and the

apparent scientific impact of a scientist– Reduces the effect of highly cited papers – But not without shortcomings: No field normalisation. H [read: age]

index.

Journal impact factor

One of the most well-known bibliometric indicators

Originally introduced by Eugene Garfield as a measure of the frequency with which the average article in a journal has been cited

Average citation rate for 1 and 2 years old articles published in a particular journal:

Number of citations received in year T to the articles published in a journal during the years T-1 og T-2 divided by the number of articles published in the journal during the years T-1 og T-2

40

Common indicators in bibliometric studies of research

Indicator Data

Research output Publications per unit

Research profile Publications per field

Scientific impact Number of citations

Research collaboration Co-authorship

Addresses are the basis for indicators of:

-Publication activity per university-Research collaboration (combinations of addresses in publications)

Map of scientific collaborations 2005 to 2009

Norwegian articles with and without international co-authorship, 1981-2009

1981

1982

1983

1984

1985

1986

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

0500

1,0001,5002,0002,5003,0003,5004,0004,5005,0005,500

Without international collaboration

With international collaboration

Kilde: Citation Report for Norway

The institutional national and international collaboration profile of the University of Bergen based on co-authorship

Fac of

Math

em &

Natu

ral Sci

Fac of

Med

icine

and D

entis

try

Fac of

Psy

chology

Fac of

Soc

ial S

cienc

es

Facult

y of H

uman

ities

Fac of

Law-5%

0%5%

10%15%20%25%30%35%40%45%50%55%60%65%70%75%80%

International National

The international collaboration profile of the University of Bergen based on co-authorship

The institutional international collaboration profile of the University of Bergen based on co-authorship

Max-Planck-Gesellschaft

Universidadede São Paulo

Copenhagen Univ

Aarhus Univ

Centre national de la recherche scientifique,

France

Helsinginyliopisto

University College London

University of Oxford

National Health Service, UK

King's College London, U. of London

Vrije UniversiteitAmsterdam

Russian Academy of Sciences

Uppsala Univ

Karolinska Inst

GothenbourghUniv

Lunds Univ

Sahlgrenska UnivHospital

Umea Univ

Stockholms Univ

University of Cape Town

Univerzita Karlova v Praze, Prague

MakerereUniversity, Uganda

Harvard University

University of California, San Diego

UiB

47

Evaluation of research performance - an important application of bibliometrics

Growing, often controversial, policy interest to use bibliometric techniques as output measures of research productivity and impact

Bibliometrics evolved to a standard tool in science policy and research evaluation

Science policy applications

Monitoring research systems

Evaluations

Funding

Applications in research evaluations

Levels of analyses– Country– Institutions– Departments– Divisions/research groups– (Individuals – anonymous presentation)

Bibliometric indicators for funding purposes

In 2004, Norway implemented a bibliometric model for performance based budgeting of research institutions– The funding of these institutions is now partially based on the

measurement of their scientific and scholarly publishing – National bibliographic database (CRIStin) used for this

purpose– Similar system for research institutes and hospitals

Questions – comments?

www.nifu.no