![Page 1: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/1.jpg)
THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION
Philip Purnell
September 2010
![Page 2: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/2.jpg)
HOW DO WE EVALUATE RESEARCH?
• Research grants– Number and value
• Prestigious awards – Nobel Prizes
• Patents– Demonstrating innovative research
• Faculty– Number of post-graduate researchers
• Citation analysis– Publication and citation counts
– Normalised by benchmarks
• Peer Evaluation– Expensive, time consuming and subjective
![Page 3: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/3.jpg)
A BRIEF HISTORY OF THE CITATION INDEX
• Concept first developed by Dr Eugene Garfield – Science, 1955
• The Science Citation Index (1963)– SCI print (1960’s)– On-line with SciSearch in the 1970’s – CD-ROM in the 1980’s– Web interface (1997) Web of Science
• Content enhanced:– Social Sciences Citation Index (SSCI)– Arts & Humanities Citation Index (AHCI)
• The Citation Index – Primarily developed for purposes of information retrieval– Development of electronic media and powerful searching tools
have increased its use and popularity for purposes of Research Evaluation
![Page 4: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/4.jpg)
WEB OF SCIENCE JOURNAL SELECTION POLICY
• Why do we select journals?
![Page 5: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/5.jpg)
WHY NOT INDEX ALL JOURNALS?
0
20
40
60
80
100
120
0 1000 2000 3000 4000 5000 6000
# of journals
% o
f dat
abas
e
Articles Citations
40% of the journals:
• 80% of the publications
• 92% of cited papers
4% of the journals:
• 30% of the publications
• 51% of cited papers
![Page 6: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/6.jpg)
HOW TO DECIDE WHICH JOURNALS TO INDEX
• Approx. 2000 journals evaluated annually
– 10-12% accepted
• Thomson Reuters editors– Information professionals
– Librarians
– Experts in the literature of their subject area
Web of Science
Journals under evaluation
Journal ‘quality’
![Page 7: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/7.jpg)
THOMSON REUTERSJOURNAL SELECTION POLICY
• Publishing Standards– Peer review, Editorial conventions
• Editorial content– Addition to knowledge in specific subject field
• Diversity– International, regional influence of authors, editors, advisors
• Citation analysis– Editors and authors’ prior work
![Page 8: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/8.jpg)
Region # Journals from Region in Web of Science
Europe 5,573 49%
North America 4,251 38%
Asia-Pacific 965 9%
Latin America 272 2%
Middle East/Africa 200 1%
Language # Journals in Web of Science
English 9114 81%
Other 2147 19%
GLOBAL RESEARCH REPRESENTATIONWEB OF SCIENCE COVERAGE
![Page 9: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/9.jpg)
Analyses based on authoritative, consistent data from the world’s leading provider of Research Evaluation solutions
Thomson Reuters has developed a selection policy over the last 50 years designed to hand-pick the relevant journals containing the core content over the full range of scholarly disciplines
This has created a large set of journals containing comparable papers and citations
Thomson Reuters has always had one consistent editorial policy to index all journals cover-to-cover, index all authors and index all addresses. This unique consistency makes Web of Science the only suitable data source for citation analysis
SUMMARYCONSISTENCY IS THE KEY TO VALIDITY
![Page 10: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/10.jpg)
GOVERNMENTS AND INSTITUTIONS USING TR DATA FOR EVALUATION (INCL.)
• Germany: IFQ, Max Planck Society, DKFZ, MDCUS
• Netherlands: NWO & KNAW
• France: Min. de la Recherche, OST - Paris, CNRS
• United Kingdom: King’s College London; HEFCE
• European Union: EC’s DGXII(Research Directorate)
• US: NSF: biennial Science & Engineering Indicators report (since 1974)
• Canada: NSERC, FRSQ (Quebec), Alberta Research Council
• Australian Academy of Science, gov’t lab CSIRO
• Japan: Ministry of Education, Ministry of Economy, Trade & Industry
• People’s Republic of China: Chinese Academy of Science
• Times Higher Education: World University Rankings (from 2010)
10
![Page 11: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/11.jpg)
EVALUATING COUNTRIES
![Page 12: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/12.jpg)
SCIENTIFIC RESEARCH IMPACT IN CENTRAL EUROPE
12Thomson Reuters InCites
![Page 13: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/13.jpg)
OUTPUT AND PRODUCTIVITYBULGARIAN RESEARCH 1998 - 2008
13
![Page 14: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/14.jpg)
COMPARATIVE IMPACT IN SELECTED FIELDSBETWEEN COUNTRIES
14Source: Thomson Reuters InCites
![Page 15: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/15.jpg)
BULGARIAN RESEARCH RELATIVE PRODUCTIVITY BY FIELD
15
22% Bulgarian papers are in Chemistry
Source: Thomson Reuters InCites
<1% Bulgarian papers are in Psychiatry
![Page 16: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/16.jpg)
EVALUATING INSTITUTIONS
![Page 17: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/17.jpg)
EVALUATING INSTITUTIONS
Source: Thomson Reuters
North America University Science Indicators
![Page 18: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/18.jpg)
CITATIONS PER PAPERMATHEMATICS
18Source: Thomson Reuters InCites
![Page 19: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/19.jpg)
COMPARISON OF TOP MATHEMATICS INSTITUTES AROUND THE WORLD
19Source: Thomson Reuters InCites
![Page 20: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/20.jpg)
WITH WHOM DOES OUR FACULTY COLLABORATE?
20Source: Thomson Reuters InCites
![Page 21: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/21.jpg)
WHICH COLLABORATIONS ARE THE MOST VALUABLE?
21
Collaborations with these institutions have produced highly cited papers within their subject fields
Source: Thomson Reuters InCites
![Page 22: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/22.jpg)
EVALUATING JOURNALS
![Page 23: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/23.jpg)
CALCULATING 2009 IMPACT FACTOR - JOURNAL OF CONTAMINANT HYDROLOGY
Citations in 2009
To items published in 2008 = 153
To items published in 2007 = 239
Sum = 392
Number of items
Published in 2008 = 97
Published in 2007 = 98
Sum = 195
392
195
= 2,01
![Page 24: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/24.jpg)
JOURNAL IMPACT FACTORSELECTED CHEMISTRY JOURNALS
24Thomson Reuters Journal Citation Reports
![Page 25: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/25.jpg)
25
USING THE IMPACT FACTOREVALUATING JOURNALS
• Appropriate use– To evaluate journals within a subject field
• Misuse– Comparison of journals from different fields
– Evaluation of individual articles
– Evaluation of institution or researcher
![Page 26: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/26.jpg)
26
USING THE IMPACT FACTOR MISUSE: EVALUATING INDIVIDUAL PAPERS
30% of articles in Food Policy were not cited at all
Journal Impact Factor = 2,01
![Page 27: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/27.jpg)
BENCHMARK YOUR PAPERS AGAINST GLOBAL AVERAGES – IS THIS A HIGHLY CITED PAPER?
27
Hematology articles from this year have been cited 18,83 times
This article is ranked in the 12,92nd percentile in its field by citations
Articles published in ‘Blood’ from 2004 have been cited 34,30 times
This paper has received 40/34,30=1,17 times the expected citations for this journal
This paper has received 40/18,83=2,12 times the expected citations for this subject category
![Page 28: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/28.jpg)
EVALUATING INDIVIDUALS
![Page 29: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/29.jpg)
HOW CAN WE COMPARE RESEARCHERS?
29
Author A: 60 papers Author B: 117 papers
Source: Thomson Reuters InCites
![Page 30: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/30.jpg)
OBTAIN MULTIPLE MEASURES
![Page 31: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/31.jpg)
RECOGNIZE THE SKEWED NATURE OF CITATION DATA
• Citation distribution is always skewed– Few highly cited papers
– Majority cited little or not at all
• Distribution type– Always distorted
– Human decision• E.g. Criticality
![Page 32: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/32.jpg)
SUMMARY (I): TREAT AS A SCIENTIFIC STUDY
• Ask whether the results are reasonable
• Follow scientific process for evaluating data
• Apply scientific skepticism
![Page 33: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/33.jpg)
SUMMARY (II): HOW DO WE EVALUATE RESEARCH?
• Research grants– Number and value
• Prestigious awards – Nobel Prizes
• Patents– Demonstrating innovative research
• Faculty– Number of post-graduate researchers
• Citation analysis– Publication and citation counts
– Normalised by benchmarks
• Peer Evaluation– Expensive, time consuming and subjective
![Page 34: THE ROLE OF CITATION ANALYSIS IN RESEARCH EVALUATION Philip Purnell September 2010](https://reader035.vdocument.in/reader035/viewer/2022062421/56649d9c5503460f94a845ee/html5/thumbnails/34.jpg)
THANK YOU
Philip Purnell
September 2010