USING BIBLIOMETRICSA GUIDE TO EVALUATING RESEARCH PERFORMANCE USING CITATION DATA

Philip PurnellLiverpool, March 2011

DO WE NEED MORE METRICS?1962

METRICS ARE HERE TO STAY!2010

• Context and limitations

• Data

• Examples

4

Context and limitations

HOW IS RESEARCH EVALUATED?

• Peer Evaluation– Reputation

• Productivity– Number of researchers– Publication output

• Grant awards– Research funding awarded

• Prestigious awards – Nobel Prizes

• Innovation– Industry income and patents

• Teaching– Academic Reputation Survey,

higher degrees

• International Mix– National / International staff

and students

• Citation analysis– Normalised for volume and

subject area

7

UNIQUE CHALLENGES IN HUMANITIESCITEDNESS

8

UNIQUE CHALLENGES IN HUMANITIESCITATION IMPACT

Impact = CitationsPapers

The Data

A BRIEF HISTORY OF THE CITATION INDEX

• Concept first developed by Dr Eugene Garfield – Science, 1955

• The Science Citation Index (1963)– SCI print (1960’s)– On-line with SciSearch in the 1970’s – CD-ROM in the 1980’s– Web interface (1997) Web of Science

• Content enhanced:– Science Citation Index – Expanded (SCI-E)– Social Sciences Citation Index (SSCI)– Arts & Humanities Citation Index (AHCI)– Conference Proceedings Citation Index (CPCI)

WHY NOT INDEX ALL JOURNALS?

0

20

40

60

80

100

120

0 1000 2000 3000 4000 5000 6000

# of journals

% o

f dat

abas

e

Articles Citations

40% of the journals:

• 80% of the publications

• 92% of cited papers

4% of the journals:

• 30% of the publications

• 51% of cited papers

HOW DO WE DECIDE WHICH JOURNALS TO INDEX?• Approx. 2.500 journals evaluated annually

– 10-12% accepted

• Thomson Reuters editors– Information professionals – Librarians – Experts in the literature of their subject area

Web of Science

Journals under evaluation

Journal ‘quality’

THOMSON REUTERSJOURNAL SELECTION POLICY

• Publishing Standards– Peer review, Editorial conventions

• Editorial content– Addition to knowledge in specific subject field

• Diversity– International, regional influence of authors, editors, advisors

• Citation analysis– Editors and authors’ prior work

Region # Journals from Region in Web of ScienceEurope 5,573 49%

North America 4,251 38%Asia-Pacific 965 9%

Latin America 272 2%Middle East/Africa 200 1%

Language # Journals in Web of ScienceEnglish 9114 81% Other 2147 19%

GLOBAL RESEARCH REPRESENTATIONWEB OF SCIENCE COVERAGE

WEB OF SCIENCESUMMARY• World’s largest citation index

– >49 millon papers (75% with citations)– Approx. 800 million citations

• Multidisciplinary– 252 categories covering Science, Social Sciences, Arts & Humanities

• 110 years consistent coverage– Publishers, societies, open access and electronic journals

• Content derived from – 12,000 scholarly journals– 120,000 conferences

• Updated weekly• Exclusively hosted on ISI Web of Knowledge platform

16

GLOBAL RESEARCH COMMUNITYUSING WEB OF SCIENCE

Asia-Pacific

353 institutions 26 countries

Europe,Middle

East andAfrica

2,500+ institutions 50 countries

244 institutions 12 countries

LatinAmerica

760 institutions2 countries

NorthAmerica

Russia 237 institutions

4.000 Research institutions 91 countries

GOVERNMENTS AND INSTITUTIONS USING TR DATA FOR EVALUATION (INCL.)•United Kingdom: King’s College London; HEFCE

•Germany: IFQ, Max Planck Society, DKFZ, MDCUS

•Netherlands: NWO & KNAW

•France: Min. de la Recherche, OST - Paris, CNRS

•Russia: Russian Foundation for Basic Research (RFBR)

•European Union: EC’s DGXII(Research Directorate)

•US: National Science Foundation (since 1974)

•Canada: NSERC, FRSQ (Quebec), Alberta Research Council

•Australian Academy of Science, gov’t lab CSIRO

•Japan: Ministry of Education, Ministry of Economy, Trade & Industry

•People’s Republic of China: Chinese Academy of Science

•University rankings agencies (THE, Shanghai, etc.)17

Authoritative data from the world’s leading provider of research evaluation data

Strict selection policy applying consistent criteria over the last 50 years

This has created a large set of journals containing comparable papers and citations

One consistent editorial policy

Unique set of multi-disciplinary comparable data

CONSISTENCY IS THE KEY TO VALIDITY

Examples

PRODUCTIVITY

20

PRODUCTIVITY – INSTITUTIONS

21

PRODUCTIVITY BY UNIVERSITY DEPARTMENT

INFLUENCE

Source: Thomson Reuters U.S. and Canadian University Science Indicators

Citations to North American papers

HOW COMPETITIVE IS OUR UNIVERSITY?

24

……

…

HOW DOES OUR UNIVERSITY COMPARE LOCALLY AND GLOBALLY?

25

H-INDEX

27

EFFICIENCY H-INDEX• Distribution based indicator

– Attempt to reflect productivity and influence– Does not control for time period, field or doc. type

Thomson Reuters InCites

SELF-CITATIONS

29

EFFICIENCYJOURNAL IMPACT FACTOR

2009 Impact Factor

200920082007

Source paper – published in 2009

Cited reference – published in 2007 or 2008

Citations

All Previous Years

2006 2010

CALCULATING 2009 JOURNAL IMPACT FACTOR HEALTH ECONOMICS

Citations in 2009To items published in 2008 = 156

To items published in 2007 = 210

Sum = 366

Number of itemsPublished in 2008 = 97

Published in 2007 = 85

Sum = 182

366

182

= 2.011

EFFICIENCYJOURNAL IMPACT FACTOR

32

INTRODUCTION TO THE IMPACT FACTOR NATURAL FLUCTUATIONS OVER TIME

CITATION BEHAVIOUR VARIES BETWEEN SUBJECT CATEGORIES

EFFICIENCY5-YEAR IMPACT FACTOR

Citation rates to journals in the ‘Acoustics’ category are

maintained thus increasing the 5-year Impact Factor relative to

its 2-Year counterpart

EFFICIENCY5-YEAR IMPACT FACTOR

The opposite picture in ‘Genetics and Heredity’ demonstrates a

shorter citation lag

USING THE IMPACT FACTOREVALUATING JOURNALS

• Appropriate use– To evaluate journals

• Misuse– Evaluation of individual articles– Evaluation of institution or researcher

38

USING THE IMPACT FACTOR MISUSE: EVALUATING INDIVIDUAL PAPERS

30% of articles in Food Policy were not cited at all

Source: Thomson Reuters Journal Citation Reports

Journal Impact Factor = 2,011

BENCHMARK YOUR PAPERS AGAINST GLOBAL AVERAGES – IS THIS A HIGHLY CITED PAPER?

39

Hematology articles from this year have been cited 18,83 times

This article is ranked in the 12,92nd percentile in its field by citations

Articles published in ‘Blood’ from 2004 have been cited 34,30 times

This paper has received 40/34,30=1,17 times the expected citations for this journal

This paper has received 40/18,83=2,12 times the expected citations for this subject category

WHICH AUTHORS’ PAPERS HAVE PERFORMED BEST IN THEIR FIELD?

40

Source: Thomson Reuters InCites

Computer Science

Medicine

WHICH ARE OUR CENTRES OF EXCELLENCE?

41

Dept. Medicine Dept. Mechanical Engineering

PERCENTILE INDICATORS

• Paper percentile– Determine percentage of papers at each level of citation

IN WHICH JOURNALS HAS OUR CHEMISTRY DEPT. PUBLISHED?

HOW HAVE THOSE PAPERS PERFORMED?

44

HOW MANY CITATIONS HAS THE CHEMISTRY DEPT. RECEIVED?

45

IN WHICH JOURNALS ARE THOSE CITATIONS PUBLISHED?

46

LIMITATIONS TO CITATION ANALYSIS

• Albert Einstein, modest paper and citation count

• « What nobler employment,or more valuable to the state,than that of the man who instructs the rising generation? »Marcus Tullius Cicero 106B.C. – 43B.C. H-Index still going up !

• High co-authorships

• Author placement

48

USING BIBLIOMETRICSA GUIDE TO EVALUATING RESEARCH PERFORMANCE USING CITATION DATA

Philip PurnellLiverpool, March 2011