Bibliometric indicators, research evaluation

and funding parameters

HEFCE Consultation Events January 2008

Henk F. Moed

Centre for Science and Technology Studies (CWTS) Leiden University, the Netherlands

Contents

1. Short introduction to citation analysis

3. Earlier studies and their outcomes

5. Effects of performance assessments upon performers’ behavior

7. Metrics, research evaluation and funding parameters

Citation Analysis in Research Evaluation

Springer, 2005, 350 pp.

Henk F. Moed

CWTS, Leiden University, the Netherlands

1.

A concise introduction to citation analysis

Topics

• Differences in citation practices among fields

• Skewness of citation distributions

• Field-normalised citation impact indicators

• Coverage of the Web of Science

• Types of bibliometric studies

• What do citations measure

Scope: Science vs. Technology

Contribution of technologies to technological progress

The influence of technology upon scientific development

Technology

The science base of technology

Contribution of science groups to scientific progress (this lecture)

Science

TechnologyScience

Influenced/CitingInfluencing/cited

Major differences in citation levels and half-lives (CHL) among research fields

0.1

1

10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

AGE

CIT

ES

/PA

PE

R

BIOCHEM &MOL BIOL

CLINNEUROLOGY

ORTHOPEDICS

PHARMACOL& PHARMACY

CHL=5.4 ± 0.1

CHL=5.0 ± 0.1

CHL=12.1 ± 0.4

CHL=8.8 ± 0.3

Geology; MineralogyOrthopedics12

Mathematics; Ecology11

Acoustics; Appl Math10

Agriculture; GeosciencesClin Neurol; Emerg Med9

Mech Eng; At Mo Ch PhysDentistry; Otorhinolaryn8

Cond Mat Phys;Chem EngAnat; Surgery; Nursing7

Astron; Org Chem;Med Chem; Neurosci6

Nanoscience & Technol; Appl Physics;

Pharmacol & Pharmacy Bioch & Mol Biol; Oncol

5

Immunol; Transplant4

Natural SciencesMedical-Biological SciCited Half Life

Normalised citation impact (1.0 = at world average)

The average citation rate of a unit’s papers

÷world citation average in the subfields

in which the unit is active

Corrects for differences in citation practices among fields,

publication years and type of article

Normalized journal impact factors

0.01

0.1

1

10

0.01 0.1 1 10 100

JCR-like IF

No

rmal

ized

IF

Mathematics

Biochem & Mol Biol

Profile of a group in Medical Pharmacology

0.0 5.0 10.0 15.0 20.0 25.0 30.0 35.0 40.0

NEUROSCIENCES (1.42)

ENDOCRIN & METAB (1.89)

BIOCH & MOL BIOL (0.92)

PHARMACOL & PHAR (1.66)

MULTIDISCIPL SC (0.52)

BEHAVIORAL SC (0.85)

PHYSIOLOGY (1.20)

GENETICS & HERED (3.74)

REPROD BIOLOGY (0.51)

CELL BIOLOGY (1.49)

OBSTETRICS & GYN (0.90)

RHEUMATOLOGY (2.20)

CLIN NEUROLOGY (1.87)

DEVELOPMENT BIOL (0.69)

UROLOGY & NEPHRO (2.61)

ZOOLOGY (0.74)

% Articles

Impact:Blue=High

Orange=AverageWhite=Low

Normalised citation impact

159 NL Academic Chemistry Departments

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

4.5

0 200 400 600 800

Total Publications

Nor

mal

ised

Cita

tion

Impa

ct

World Average

Skewed citation distribution of 2 journals

0.01

0.1

1

10

100

1 10 100

Nr Cites

% P

aper

s

ANALYT CHEMANALYT CHIM ACTA

N=1,932Mean=4.5

%Uncited=12

N=1,466Mean=1.9

%Uncited=28

A skewed citation distribution of a research groups’s papers is a normal phenomenon

0

2

4

6

8

10

12

14

16

18

20

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40

Nr Cites

% P

aper

s

N=150Mean=38.5Skewn=2.6

N=145Mean=4.8

Skewn=1.6

N=174Mean=5.5

Skewn=4.6

72280

Skewed citation distributions: ‘Oeuvre-hypothesis’

• Articles are elements from publication oeuvres of groups carrying out a research programme

• Authors citing an oeuvre/programme tend to cite ‘key’ or ‘flag’ papers from that oeuvre

• Key papers may snatch away citations from the oeuvre’s other articles

WoSNon-WoS

Non-WoS WoS

Citing/Source

Cited/Target

?%?%

Measurement of internal WoS Coverage

Non-Wos Journals

Books

Conference proceedings

Reports

Etc.

ZUCKERMAN H, SCIENTOMETRICS, v 12, p 329, 1987

ROUSSEAU R, SCIENTOMETRICS, v 43, p 63, 1998

MERTON RK, ISIS, v 79, p 606, 1988

GILBERT GN, SOC STUDIES SCI, v 7, p 113, 1977

GARFIELD E, ESSAYS INFORMATION S, v 8, p 403, 1985

GARFIELD, E. CITATION INDEXING, 1979 (BOOK!)

ABT HA, J AM SOC INF SCI T, v 53, p 1106, 2004

SCIENTOMETRICS 60 (3): 295-303, 2004

In basic science the percentage of 'authoritative' references decreases as bibliographies become shorter

Moed, HF; Garfield, E.

Y

Y

Y

Y

N

N

Y

Y

in WO

S

RF

SO

TI

AU

WoS Coverage = 5/7 = 71%

Not in WoS

Overall WoS coverage by main field

Humanities & Arts

Other Soc SciSoc Sci ~ MedicinePhys & Astron

MODERATE (<40 %)GeosciencesClin Medicine

EngineeringPsychol & PsychiatChemistry

EconomicsBiol Sci – Anim & Plants

Biol Sci – Humans

MathematicsAppl Phys & ChemBiochem & Mol Biol

GOOD(40-60%)VERY GOOD (60-80%)EXCELLENT (> 80%)

WoSNon-WoS

Non-WoS WoS

Citing/Source

Cited/Target

Three types of citation analysis

2. Target Expanded

3. Source Expanded

1. Pure WoS

4 Types of bibliometric studies

ModerateNo citation analysis at all4

Good – Moderate

WoS+non WoSWoS+non WoSSource expanded

3

Very Good – Good

WoSWoS+non WoSTarget expanded

2

Excellent – Very Good

WoSWoS‘Pure’ WoS1

WoS coverage

Citing/SourceCited/TargetType

What do citations measure?

• Many studies showed positive correlations between citations and qualitative judgments

• In principle it is valid to interpret citations in terms of intellectual influence

• But the concepts of citation impact and intellectual influence do not coincide

2.

Some earlier studies and their outcomes

16 broad disciplines

Social Sci ~ MedicineSOC-MEDEngineeringENG

Social Sci(SOC)Economics & BusinessECON

Psychol & PsychiatPSYClinical MedicineCLM

Physics & AstronPHYSChemistryCHEM

Multi-disciplinary(MULTI)Biol Sci ~ HumansBIOL-HU

Mol Biol & BiochemMOLBBiol Sci ~ Anim & Plants

BIOL-A&P

MathematicsMATHAppl Phys & ChemAPC

GeosciencesGEOArts & Humanities(A&H)

General European Univ

TOP 25%BOTTOM 25%

TOP 25%

BOTTOM 25%

Impact

Publications

(A&H)

APC

BIOL-AP

BIOL-HU

CHEM

CLM ECON

ENG

GEO

MATH

MOLB

(MULTI)

PHYS

PSY

SOC-MED

(SOC)

0

25

50

75

100

0255075100PUBLICATION RANK PTCL

CIT

AT

ION

I P

AC

T R

AN

K P

CT

L

Among top 25 % in publication output and citation impact

‘Top’ US/UK research university

(SOC)

SOC-MEDPSY

PHYS (MULTI) MOLB

MATH

GEO

ENG ECON

CLM

CHEM

BIOL-HUBIOL-AP

APC

(A&H)

0

25

50

75

100

0255075100PUBLICATION RANK PTCL

CIT

AT

ION

I P

AC

T R

AN

K P

CT

L

University has a top position

in each discipline

Univ Milano Univ Minnesota – Minnea.15

Univ Michigan - Ann Arbor Osaka Univ 14

Erasmus Univ Rotterdam Univ Penn 13

Univ Wien Univ Oxford 12

Univ Pittsburgh Stanford Univ 11

Univ Calif Los Angeles Univ Coll London 10

Univ Washington - Seattle Johns Hopkins Univ 9

Univ Tokyo Univ Cambridge 8

Univ Penn Kyoto Univ 7

Univ Calif San Francisco Univ Michigan - Ann Arbor 6

Karolinska Inst Stockholm Univ Washington - Seattle 5

Univ Toronto Univ Calif Los Angeles 4

Johns Hopkins Univ Univ Toronto 3

Harvard Univ Univ Tokyo 2

Univ Texas - Houston Harvard Univ 1

OncologyAll fieldsRank

1.74

1.91

1.35

1.22

1.26

1.01

1.52

0.90

1.15

1.49

1.22

1.43

1.61

1.69

0.90

1.26

1.26

0.72

1.47

1.50

1.13

1.31

1.07

2.10

1.43

1.00

1.21

1.08

1.68

2.02

1.85

2.65

1.63

1.33

10% 8% 6% 4% 2% 2% 4% 6% 8% 10%

BIOCHEM&MOL BIOL

NEUROSCIENCES

VETERINARY SC

RAD,NUCL MED IM

MEDICINE,GEN&INT

CLIN NEUROLOGY

IMMUNOLOGY

SURGERY

ONCOLOGY

GENETICS&HEREDIT

CARD&CARDIOV SYS

CHEM,MULTIDISC

CELL BIOLOGY

PSYCHIATRY

MULTIDISCIPL SC

PHARMACOL&PHARMA

PLANT SCIENCES

UNIV ZURICH UNIV COLL LONDON

% Articles

Impact:Black=High

Dashed=AverageBlank=Low

Normalised citation impact

ZURICH

BASEL

BERN

EPFLETHZ

GENEVE

LAUSANNE

AMSTERDAM

CAMBRIDGE

EDINBURGH

FREIBURG

HEIDELBERG

HELSINKIKAROLINSKA

LEIDENLEUVEN

LUND

MILANO

MUNCHEN

OXFORD

PARIS VI

PARIS XISTRASBOURG I UTRECHT

BERKELEY

FU BERLIN

HU BERLINKCL

KOBENHAVNLYON I

MELBOURNE

MICHIGAN

OSLO

PENN

TORONTO

UPPSALAWIEN

WURZBURG

0.8

1.0

1.2

1.4

1.6

1.8

2.0

0 50 100 150 200 250

NR ARTICLES IN WORLD TOP 5 %

AC

TU

AL

/EX

PE

CT

ED

NR

AR

TS

IN W

RL

D T

OP

5 %

2.5Biol Sci ~ Humans

3.

Effects of performance assessments upon

performers’ behavior

Topics

3. The UK Research Assessment Exercises (RAE)

Timing effects and shifts in criteria in UK Research Assessment Exercises (RAE)

-4.0

-3.0

-2.0

-1.0

0.0

1.0

2.0

3.0

4.0

86 87 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03

Years

An

nu

al G

row

th R

ate

% UK Articles% UK Authors

1995/61992 2000

Total Publication

Counts

Shift from Quantity to

Quality

ResearchActiveStaff

% UK authors

% UK Articles

Can one increase actual citation impact by…..

• Increasing author self citation?

• Publishing in high impact journals?

• Collaborate more intensively?

• Publishing with US authors because they overcite their own papers?

• Publishing less, only the very best papers?

• Making citation arrangements?

At the level of research groups, actual citation impact and journal prestige tend to show only weak correlations

[Set of 2,150 UK authors with > 10 articles per year]

23 %Normalised journal impact / prestige

11 %Average journal impact factor

0 %No published articles

Explained variance in actual citation impact

Indicator

More collaboration M higher impact?

• Some studies report positive correlations between a paper’s number of authors and its citation impact

• They ignore differences among fields

• It all depends upon who collaborates with whom

• Causality issue: ‘Good’’ research may lead to collaboration

Do US scientists overcite papers from their own country?

• The crucial issue at stake is the adequacy of the norm against which referencing practices of US scientists is evaluated

• A first study found no conclusive evidence that US scientists in science fields excessively cite papers originating from their own country (Moed, 2005)

Publishing less P Higher impact?

• One would expect a higher citation impact per paper (crown indicator)

• But what are the longer tem effects?

• PhD students need papers in their CV’s

• Relationship between a research group’s ‘bricks’ and ‘flag’ papers is complex

Mutual citation arrangements?

• A high impact group receives its citations from dozens if not hundreds of institutions

• The distribution of citations amongst citing institutions is very skewed

• The contribution of the tail is very large

• Making arrangements with a few institutions will not help much

Why use sophisticated citation analysis in RAE?

• Shift in focus from quantity to quality

• Reduction unintended effects of using less sophisticated indicators

• Use of absolute rather than relative standards

• Formal rather than informal use (transparency)

4.

Metrics, research evaluation and funding parameters

Metrics and funding parameters

• Policy level: Central vs. institutional (e.g., national vs. university)

• Elaboration of indicator data: Statistical vs. evaluative

Data ElaborationFunding

parameters

Central

Institution

Indicatorsof groups / individuals

Aggregate per

institution

Combine with peer

review

Acrossinstitutions

Within aninstitution

Metrics and funding parameters

Use of metrics in allocation of research funds

• At a central level: To distribute funds across institutions based on aggregate statistics

• At an institutional level: Combined with peer review to evaluate groups and individuals; outcomes are used to distribute funds within an institution

Aggregate statistics

• Random errors to some extent cancel out

• Identify and neutralize systematic errors

END