TIPS WHEN USING BIBLIOMETRICS

UNITED KINGDOM

OCTOBER 2010

USING BIBLIOMETRICS IN CITATION ANALYSIS

• The Development of Publication and Citation Analysis

• Bibliometrics and Peer Judgment: A Two-Pronged Approach

ONE: CONSIDER WHETHER AVAILABLE DATA CAN ADDRESS THE QUESTION

Large data sets (nations, regions, large institutions)

Engineering & Applied Sciences

Fundamental &Basic Sciences

Small data sets(small departments, individual

researchers)

LowConfidence

ModerateConfidence

ModerateConfidence

HighConfidence

Social Sciences

Individual

Medical

Country

TWO: CHOOSE PUBLICATION TYPES, FIELD DEFINITIONS AND YEARS OF DATA

• Publication types– Research Articles

– Notes

– Review articles

• Field definitions– Difficulties with characterization

• Time frame– Citation lag

THREE: DECIDE ON WHOLE OR FRACTIONAL COUNTING

68 authors

17 institutions

18 citations

FOUR: JUDGE WHETHER DATA REQUIRE EDITING TO REMOVE ARTIFACTS• Artifacts of variation

– Institution names• Heidelberg Univ, Heidelberg Univ Clin, Heidelberg Univ Educ,

Heidelberg Univ Hosp, Heidelberg Univ I, Heidelberg Univ II, Univ Heidelberg, Univ Heidelberg BZH, Univ Heidelberg Cardiol, Univ Heidelberg Clin, Univ Heidelberg Hosp, Univ Heidelberg Hosp & Clins, Univ Heidelberg J5, Univ Heidelberg Klinikujm, Univ Heidelberg Klinikum, Univ Heidelberg Mannheim, Univ Heidelberg Miriam Roehrig, Univ Heidelberg Neuebheimer Feld, Univ Heidelberg ZMBH, Univ Heidelburg, Univ Heilderberg, Univ Hosp Heidelberg, Univ Hosp Heidelberg Mannheim, Univ Klin Heidelberg, Univ Klin Heidelburg, Univ Klinikum Heidelberg, Univ Klinikum Heidelbert, Univ Klinikum Heidelburg, Univ Klinikum Heiderlberg

– Author names

• Is ROTHWELL N different from ROTHWELL NJ?• LEE S returns 14,022 publications; SUZUKI T (21,462)

…ARTIFACTS contd.COMMON OBJECTIONS TO CITATION COUNTS

• Negative citations

• Over citations– Reviews

– Methods

• Self-citation and citation circles

FIVE: COMPARE LIKE WITH LIKE

Differences in citation curves at the category level

0%

2%

4%

6%

8%

10%

12%

2006 2005 2004 2003 2002 2001 2000 1999 1998 1997Cited year

% o

f to

tal c

itat

ion

s to

th

e ca

teg

ory

Cell Biol (5.9)

Med, Gen Int (7.1)

Math (>10)

Multidisc (7.6)

Econ (>10)

Education(8.3)

Impact Factor

FIVE: COMPARE LIKE WITH LIKE

• Different citation behaviour between categories

• Even within the same category:– Age of paper

– Age and stage of author’s career

SIX: USE RELATIVE MEASURES – NOT JUST ABSOLUTE COUNTS

• Absolute counts– Total publications

– Papers published per year on average

– Papers published in top journals (various definitions)

– Total citations received

• Relative counts– Citations per paper compared with field average

– Citations versus expected citations

– Cited vs un-cited papers compared with average

– Rank within field or peer group

EIGHT: RECOGNIZE THE SKEWED NATURE OF CITATION DATA

• Citation distribution is always skewed– Few highly cited papers

– Majority cited little or not at all

• Distribution type– Always distorted

– Human decision• E.g. Criticality

TREAT CITATION ANALYSIS AS A SCIENTIFIC STUDY

• Are the data collected relevant to the question?

• Are the results reasonable?

• Follow scientific process for evaluating data

• Apply scientific skepticism