citation analysis in research evaluation (published by springer, july 2005)
DESCRIPTION
Citation Analysis in Research Evaluation (Published by Springer, July 2005). Henk F. Moed CWTS, Leiden University. Contents. Introduction: Potentialities of bibliometrics Adequacy of WoS coverage per main discipline - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/1.jpg)
Citation Analysis in Research Evaluation
(Published by Springer, July 2005)
Henk F. Moed
CWTS, Leiden University
![Page 2: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/2.jpg)
2
Contents
• Introduction: Potentialities of bibliometrics
• Adequacy of WoS coverage per main discipline
• Effects of the use of bibliometric indicators upon publication and citation practices
• Citation analysis and peer review
![Page 3: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/3.jpg)
3
Book is written for a broad audience
1. Scientists and scholars subjected to citation analysis
2. Research policy makers and managers
3. Members of peer review committees and other evaluators
4. Practitioners and students in science studies, library & information science
![Page 4: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/4.jpg)
4
Book Structure
Part No Part Title
Executive Summary (7 pp)
Part 1 General Introduction and Main Conclusions (60 pp)
Part 2 Empirical and Theoretical Chapters (250 pp)
![Page 5: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/5.jpg)
5
Part No Part Title
2.1 Assessing basic science research departments and scientific journals
2.2 The ISI Citation Indexes
2.3 Assessing social sciences and humanities
2.4 Accuracy aspects
2.5 Theoretical aspects
2.6 Citation analysis and peer review
2.7 Macro studies
2.8 New developments
![Page 6: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/6.jpg)
6
Main objective
• A firm political or societal basis for ‘basic’ science can be maintained only by further developing a system of internal quality control and performance enhancement
• This book aims at showing that citation analysis is a useful tool in such a system
![Page 7: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/7.jpg)
7
General approach: uses and limits
• The book shows that the application of citation analysis in research evaluation has reached a high level of sophistication
• It discusses numerous issues raised by scientists, journal editors and policy makers
• It shows how such issues can in principle be accounted for or solved technically
![Page 8: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/8.jpg)
8
The use of citation analysis in research evaluation is more appropriate the more it is:
• Formal
• Open
• Scholarly founded
• Supplemented with expert knowledge
• Carried out in a clear policy context with clear objectives
• Stimulating users to explicitly state basic notions of scholarly quality
• Enlightening rather than formulaic
![Page 9: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/9.jpg)
Macro Trends in the Global Science System
![Page 10: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/10.jpg)
10
Fractional counting scheme for a paper with 4 addresses
Addresses Country Frac-tion
Univ Granada, Dept A. Spain 1/4
Univ Granada, Dept B. Spain 1/4
Univ Leiden, Dept C. Netherlands 1/4
Kathol Univ Leuven, Dept D. Belgium 1/4
Spain: 0.5; Netherlands: 0.25; Belgium: 0.25
![Page 11: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/11.jpg)
11
Normalised citation impact
The average citation rate of a unit’s papers
÷World citation average in the subfields in
which the unit is active
( >1.0 : above world average)
![Page 12: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/12.jpg)
12
0
5
10
15
20
25
30
35
40
YEAR
% P
AP
ER
S (
FR
AC
TIO
NA
L) US
CN
JP
UK
DE
FR
CA
IT
ES
IN
% US papers declined
![Page 13: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/13.jpg)
13
0
1
2
3
4
5
6
7
8
9
10
80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03 04 05 06
YEAR
% P
AP
ER
S (
FR
AC
TIO
NA
L)
US
CN
JP
UK
DE
FR
CA
IT
ES
IN
China No. 2 in 2006
JP, UK, DE, FR, CA decline
Spain increases
![Page 14: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/14.jpg)
14
Convergent trend in citation impact because of globalisation
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
1.6
1.8
80-82
81-83
82-84
83-85
84-86
85-87
86-88
87-89
88-90
89-91
90-92
91-93
92-94
93-95
94-96
95-97
96-98
97-99
98-00
99-01
00-02
01-03
02-03
03-05
04-06
Publication and Citation Years
No
rma
lis
ed
Cit
ati
on
Im
pa
ct
US
UK
CA
DE
FR
IT
ES
JP
CN
INImpact Spain >1.0 in 2004/6
![Page 15: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/15.jpg)
15
Conclusion
• Assessment of a country’s performance should be carried out from a comparative, global perspective
![Page 16: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/16.jpg)
Bibliometric Rankings of World Universities
Available at: www.cwts.nl/hm
Henk F. MoedCWTS, Leiden University
The Netherlands
CWTS Graduate Course 2006
![Page 17: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/17.jpg)
17
Research universities
• Conduct research at a certain scale
This study:
• Universities with > 5,000 published articles during 1977-2004 (> 625 per year)
• With more than about 600 active researchers
![Page 18: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/18.jpg)
18
Distribution of universities among regions
0.0
0.5
1.0
1.5
2.0
2.5
0 1,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000
ARTICLES / YEAR
NO
RM
AL
ISE
D C
ITA
TIO
N IM
PA
CT
USA
EUROPE
OTHER=> 9,600
Universities with > 5,000 papers during 1997-2004 (n=386)
![Page 19: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/19.jpg)
19
Distribution of universities among regions
0.0
0.5
1.0
1.5
2.0
2.5
0 1,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000
ARTICLES / YEAR
NO
RM
AL
ISE
D C
ITA
TIO
N IM
PA
CT
USAEUROPEOTHERNL
=> 9,600
Universities with > 5,000 papers during 1997-2004 (n=386)
![Page 20: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/20.jpg)
20
16 broad disciplines
(A&H) Arts & Humanities GEO Geosciences
APC Appl Phys & Chem MATH Mathematics
BIOL-A&P
Biol Sci ~ Anim & Plants
MOLB Mol Biol & Biochem
BIOL-HU
Biol Sci ~ Humans (MULTI) Multi-disciplinary
CHEM Chemistry PHYS Physics & Astron
CLM Clinical Medicine PSY Psychol & Psychiat
ECON Economics & Business (SOC) Social Sci
ENG Engineering SOC-MED Social Sci ~ Medicine
![Page 21: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/21.jpg)
21
General European Univ
TOP 25%BOTTOM 25%
TOP 25%
BOTTOM 25%
Impact
Publications
(A&H)
APC
BIOL-AP
BIOL-HU
CHEM
CLM ECON
ENG
GEO
MATH
MOLB
(MULTI)
PHYS
PSY
SOC-MED
(SOC)
0
25
50
75
100
0255075100PUBLICATION RANK PTCL
CIT
AT
ION
I P
AC
T R
AN
K P
CT
LAmong top 25 % in publication output and citation impact
![Page 22: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/22.jpg)
22
‘Top’ US/UK research university
(SOC)
SOC-MEDPSY
PHYS (MULTI) MOLB
MATH
GEO
ENG ECON
CLM
CHEM
BIOL-HUBIOL-AP
APC
(A&H)
0
25
50
75
100
0255075100PUBLICATION RANK PTCL
CIT
AT
ION
I P
AC
T R
AN
K P
CT
L
University has a top position
in each discipline
![Page 23: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/23.jpg)
23
Conclusion
• The best European universities are among the top 25% in the world
• But their top is less broad than that of the best US research universities
• In European national academic systems top research is more evenly distributed among universities
![Page 24: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/24.jpg)
24
Policy issue
• Concentration model (US, UK) versus distributed model
• Is a concentration model a proper policy incentive in the European Research Area?
![Page 25: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/25.jpg)
25
BIGGEST 100 WORLD UNIVERSITIES (Calero et al., 2006)
USA
Europe
Japan
Univ Harvard
Univ TokyoUniv Tokyo
Univ Toronto
Univ Melbourne
Univ Sydney
Univ Queensland
![Page 26: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/26.jpg)
26
BIGGEST 100 EUROPEAN UNIVERSITIES (Calero et al, 2006)
The Netherlands
Germany
United Kingdom
Scandinavia
Italy
Switzerland
France
Spain
BelgiumOxford
Univ Munich Katholieke Univ Leuven
Karolinska Inst.ETH Zurich
Univ Milano
Univ Antwerpen
![Page 27: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/27.jpg)
27
Bottom-up Approach: 159 NL Academic Chemistry Groups
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
0 200 400 600 800
Total Publications
Nor
mal
ised
Cita
tion
Impa
ct
![Page 28: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/28.jpg)
Did global scientific publication productivity increase during the
1980s and 1990s?
A Meta Analysis of the ISI Database
![Page 29: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/29.jpg)
29
Research questions
• Did authors publish in recent years more papers than they did in the past?
• How did scholarly collaboration develop?
• Are there traces of “publication pressures” in the ISI database?
![Page 30: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/30.jpg)
30
Absolute numbers
1980 2002 Growth rate
Papers 478,000 814,000 + 70 %
Unique publishing authors
545,000 1,095,000 + 101 %
Authorships 1,187,000 3,081,000 + 160 %
![Page 31: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/31.jpg)
31
Three indicators
Indicator Interpretation
Authorships per paper (Formal) scientific collaboration
Authorships per publishing author
Average number of papers in an author’s publication list
Papers per unique publishing author
‘Overall’ author publication productivity
![Page 32: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/32.jpg)
32
Patterns in total ISI database
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02
Years
Rat
io
Authorships/Publ
Authorships/Publ Author
Publ / PublAuthor
Authorships/Paper (+52%)
Authorships/Author (+29%)
Papers/Unique Publishing Author (-15%)
Author Produc-
tivity
Collabo-ration
PapersPublished by Average
Author
![Page 33: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/33.jpg)
33
Did authors publish in recent years more papers than they did in the past?
• Answer: Yes and No
• Yes: Number of ISI articles in an average author’s annual publication list increased
• No: Number of ISI articles per unique author, published by total collection of unique publishing authors, (slightly) declined
![Page 34: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/34.jpg)
34
Conjecture – 1
If publication pressure is the dominant factor:
• Scholars increased their publication counts by collaborating more intensively – or through authorship inflation - , rather than by collectively publishing more papers
![Page 35: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/35.jpg)
Adequacy of ISI/WoS coverage
![Page 36: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/36.jpg)
36
All literature
‘Important’ literature
WoS
WoS Coverage (%) = ?
![Page 37: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/37.jpg)
37
Selection of WoS source journals
The real problem is to “make the coverage as complete as possible by expanding it beyond the core of journals whose importance to a given field is obvious” (Garfield 1979)
![Page 38: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/38.jpg)
38
How is the core expanded?
• Garfield’s criterion: The frequency at which journals are cited in the source journals that are already included in the index
• Assumption: The number of times a journal’s items are cited is an expression of its utility as communication medium
![Page 39: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/39.jpg)
39
Adequacy of ISI coverage: Two approaches
External Compare ISI database with other databases (e.g., CAS, PUBMED, or scholars’ publication lists, annual reports)
Internal Examine the extent to which the documents cited in ISI source articles are themselves published in ISI source journals
![Page 40: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/40.jpg)
40
WoSNon-WoS
Non-WoS WoS
Citing/Source
Cited/Target
?%?%
Measurement of internal WoS Coverage
Non-Wos Journals
Books
Conference proceedings
Reports
Etc.
![Page 41: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/41.jpg)
41
AU Moed, HF; Garfield, E. in WO
S
TI In basic science the percentage of 'authoritative' references decreases as bibliographies become shorter
SO SCIENTOMETRICS 60 (3): 295-303, 2004 Y
RF ABT HA, J AM SOC INF SCI T, v 53, p 1106, 2004 Y
GARFIELD, E. CITATION INDEXING, 1979 (BOOK!) N
GARFIELD E, ESSAYS INFORMATION S, v 8, p 403, 1985 N
GILBERT GN, SOC STUDIES SCI, v 7, p 113, 1977 Y
MERTON RK, ISIS, v 79, p 606, 1988 Y
ROUSSEAU R, SCIENTOMETRICS, v 43, p 63, 1998 Y
ZUCKERMAN H, SCIENTOMETRICS, v 12, p 329, 1987 Y
WoS Coverage = 5/7 = 71%
Not in WoS
![Page 42: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/42.jpg)
42
WoSNon-WoS
Non-WoS WoS
Citing/Source
Cited/Target
75%25%
Total ISI/WoS Database
![Page 43: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/43.jpg)
43
WoSNon-WoS
Non-WoS WoS
Citing/Source
Cited/Target
92%8%
Molecular Biol & Biochem
![Page 44: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/44.jpg)
44
WoSNon-WoS
Non-WoS WoS
Citing/Source
Cited/Target
29%71%
Social Sciences
![Page 45: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/45.jpg)
45
Overall ISI coverage by main field
EXCELLENT (> 80%) VERY GOOD (60-80%) GOOD(40-60%)
Biochem & Mol Biol Appl Phys & Chem Mathematics
Biol Sci – Humans Biol Sci – Anim & Plants
Economics
Chemistry Psychol & Psychiat Engineering
Clin Medicine Geosciences MODERATE (<40 %)
Phys & Astron Soc Sci ~ Medicine Other Soc Sci
Humanities & Arts
![Page 46: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/46.jpg)
46
Sub-disciplines (non-exhaustive list)
Social sciences related to medicine and health
Other social sciences
Humanities
Public environm & occupat health
Nursing
Sport sciences
Substance abuse
Sociology
Anthropology
Educational sciences
Political science
Law
Literature
Linguistics
Historical sc
Philosophy
![Page 47: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/47.jpg)
47
Important factors
Mathematics CompuMath included as from 1993 only
Preprints
Engineering & Applied sciences
Conference proceedings
Reference works
Other Social Sciences Books
Language barriers
Humanities and Arts Books
Language barriers
References to study object included
![Page 48: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/48.jpg)
48
WoSNon-WoS
Non-WoS WoS
Citing/Source
Cited/Target
Three types of studies
2. Target Expanded
3. Source Expanded
1. Pure WoS
![Page 49: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/49.jpg)
49
4 Types of bibliometric studies
Cited/Target Citing/Source ISI coverage Field/Study
1 ISI ISI Excellent – Very Good
Astronomy
2 ISI+non ISI ISI Very Good – Good
Mathematics
3 ISI+non ISI ISI+non ISI Good – Moderate
Computer Science
4 No citation analysis at all Moderate NL Law
![Page 50: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/50.jpg)
50
Social Sciences and Humanities
• It cannot be taken for granted that the ISI Citation Indexes provide such indicators in all subfields of these domains of scholarship
• A challenge would be to systematically explore alternative data sources and methodologies
![Page 51: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/51.jpg)
Effects of the use of bibliometric indicators upon publication and
citation practices
![Page 52: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/52.jpg)
52
Effects of editorial self-citations upon journal impact factors
[Reedijk & Moed, J Doc, to be publ, 2007]
• Focus on ‘consequences’ rather than ‘motives’
• Case studies
• Overall pattern
![Page 53: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/53.jpg)
53
Editorial self-citations
• A journal editor cites in his editorials papers published in his own journal
• Or he cites papers in other journals published by the same publisher
![Page 54: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/54.jpg)
54
Problem with ISI/JCR Journal Impact Factor
Citations to “citable” and “non-citable” items
÷Number of “citable” items
(Citations to letters, editorials and other “non-citable” items are “free”)
![Page 55: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/55.jpg)
55
Case: ISI/JCR Impact Factor of a Gerontology Journal (published in the journal itself)
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
2000 2001 2002 2003 2004
IMPACT FACTOR YEAR
CIT
ES
PE
R 'C
ITA
BL
E' D
OC
![Page 56: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/56.jpg)
56
Decomposition of the IF of a Gerontology journal
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
2000 2001 2002 2003 2004
IMPACT FACTOR YEAR
CIT
ES
PE
R 'C
ITA
BL
E' D
OC
Cites from journal's own editorials
Cites to 'non-citable' docs
Base score
![Page 57: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/57.jpg)
UK Research Assessment Exercises:
Informed judgments on research
quality or quantity?
Henk F. Moed
Centre for Science and Technology Studies (CWTS)
Leiden University, the Netherlands
![Page 58: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/58.jpg)
58
Key policy issues
• Are research policies aimed to enhance scientific excellence effective?
• Are evaluation processes aimed to identify scientific excellence successful?
![Page 59: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/59.jpg)
59
Effects of evaluation processes upon scientists’ “practices”
• Intended: Scientists perform better
• Unintended: Scientists seek to artificially increase their ‘ratings’
![Page 60: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/60.jpg)
60
Timing effects and shifts in criteria in UK Research Assessment Exercises (RAE)
-4.0
-3.0
-2.0
-1.0
0.0
1.0
2.0
3.0
4.0
86 87 88 89 90 91 92 93 94 95 96 97 98 99 00 01 02 03
Years
An
nu
al G
row
th R
ate
% UK Articles% UK Authors
1995/61992 2000
Total Publication
Counts
Shift from Quantity to
Quality
ResearchActiveStaff
% UK authors
% UK Articles
![Page 61: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/61.jpg)
61
At the level of research groups, actual citation impact and journal prestige tend to show only weak correlations
[Set of 2,150 UK authors with > 10 articles per year]
Indicator Explained variance in actual citation impact
No published articles 0 %
Average journal impact factor
11 %
Normalised journal impact / prestige
23 %
![Page 62: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/62.jpg)
62
Why use sophisticated citation analysis
in RAE 2008?
• Shift in focus from quantity to quality
• Reduction unintended effects of using less sophisticated indicators
• Use of absolute rather than relative standards
• Formal rather than informal use (transparency)
![Page 63: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/63.jpg)
Citation Analysis and Peer Review
Henk F. Moed
Citation Analysis in Research Evaluation
Springer, 2005
![Page 64: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/64.jpg)
64
Citation Analysis and Peer Review
[Moed, Citation Analysis in Research Evaluation, Springer, 2005]
A. Tools for peers to assess research quality of research groups in (basic) science
B. Tools for policy makers to assess peer review processes
C. Tools for peers and policy makers to address complex, general, global issues (macro/meta studies)
![Page 65: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/65.jpg)
65
Affinity applicants – Committee
0 Applicants are/were not member of any Committee
1 Co-applicant is/was member of a Committee, but not of the one evaluating
2 First applicant is/was member of a Committee, but not of the one evaluating
3 Co-applicant is member of the Committee(s) evaluating the proposal
4 First applicant is member of the Committee(s) evaluating the proposal
![Page 66: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/66.jpg)
66
For 15 % of applications an applicant is a member of the evaluating Committee (Affinity=3, 4)
0
10
20
30
40
50
60
70
% A
PP
LIC
AT
ION
S
AFFINITY APPLICANTS-COMMITTEE
Projects 63.2 10.2 11.5 5.9 9.1
0 1 2 3 4
![Page 67: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/67.jpg)
67
Probability to be granted increases with increasing affinity applicants-Committee
30
40
50
60
70
80
% G
RA
NT
ED
AP
PL
ICA
TO
NS
AFFINITY APPLICANTS-COMMITTEE
Projects 37.0 46.9 60.1 62.6 74.0
0 1 2 3 4
![Page 68: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/68.jpg)
68
Logistic regression analysis: Affinity Applicant-Committee has a significant
effect upon the probability to be granted
MAXIMUM-LIKELIHOOD ANALYSIS-OF-VARIANCE TABLE (N=2,499) Source DF Chi-Square Prob ------------------------------------------------------------- INTERCEPT 1 18.47 0.0000 Publ Impact applicant 3 26.97 0.0000 ** Rel transdisc impact applicant 1 0.29 0.5926 Affinity applicant-Committee 2 112.50 0.0000 ** Sum requested 1 45.47 0.0000 ** Institution applicant 4 25.94 0.0000 ** LIKELIHOOD RATIO 199 230.23 0.0638
![Page 69: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/69.jpg)
69
Conclusion
• The future of research assessment exercises lies in the intelligent combination of metrics and peer review
![Page 70: Citation Analysis in Research Evaluation (Published by Springer, July 2005)](https://reader034.vdocument.in/reader034/viewer/2022051401/56814033550346895dab9505/html5/thumbnails/70.jpg)
70
END