the role of bibliometrics in research management ... it’s complicated !
DESCRIPTION
The role of bibliometrics in research management ... It’s complicated !. Dr . Thed van Leeuwen Center for Science & Technology Studies (CWTS) Erasmus Seminar “Research Impact & Relevance” , June 17 th 2014. Outline. Bibliometrics and research management context - PowerPoint PPT PresentationTRANSCRIPT
The role of bibliometrics in research management ...
It’s complicated !
Dr. Thed van LeeuwenCenter for Science & Technology Studies (CWTS)
Erasmus Seminar “Research Impact & Relevance”, June 17th 2014
2
Outline
• Bibliometrics and research management context
• Infamous bibliometric indicators
• Scientific integrity and misconduct
• Applicability of bibliometrics
• Take-home messages …
3
Bibliometrics and the research management context
What is bibliometrics ?
• Bibliometrics can be defined as the quantitative analysis of science and technology (development), and the study of cognitive and organizational structures in science and technology.
• Basic for these analyses is the scientific communication between scientists through (mainly) journal publications.
• Key concepts in bibliometrics are output and impact, as measured through publications and citations.
• Important starting point in bibliometrics: scientists express, through citations in their scientific publications, a certain degree of influence of others on their own work.
• By large scale quantification, citations indicate (inter)national influence or (inter)national visibility of scientific activity, but should not be interpreted as synonym for ‘quality’.
‘Classical’ image of the Credibility cycle
Credibility cycle (adapted from Latour and Woolgar (1979) & Rip (1990)
PEER REVIEW
Rise of performance indicators & bibliometrics
• Need for formalised measures• ‘Push’ from science policy (from 1970s onwards)
• Independent of peer review
• New Public Management / Neo-liberalism (from 1980s onwards)
• Matrix-structure science (Whitley)
• Researchers part of international community
Peer review
• But also part of local institutions
Specific management practices (appraisals, external evaluations)
• Institute managers not always part of international expert community
• Tighter forms of management (from the 1990s onwards)
Distance
Extended credibility cycle
8
Infamous bibliometric indicators:JIF & H-index
Definitions of Journal Impact Factor & Hirsch Index
• Definition of JIF:
– The mean citation score of a journal, determined by dividing all
citations in year T by all citable documents in years T-1 and T-2.
• Definition of h-index:
– The ‘impact’ of a researcher, determined by the number of received
citations of an oeuvre, sorted by descending order, where the number
of received citations on that single paper equals the rank position.
Problems with JIF• Methodological issues
– Was/is calculated erroneously (Moed & van Leeuwen, 1996)
– Not field normalized
– Not document type normalized
– Underlying citation distributions are highly skewed (Seglen, 1994)
• Conceptual/general issues
– Inflation (van Leeuwen & Moed, 2002)
– Availability promotes journal publishing
– Is based on expected values only
– Stimulates one-indicator thinking
– Ignores other scholarly virtues
Deconstructing the myth of the JIF…• Take the Dutch output
• Similar journal impact classes• Focus on publications that belong to the top 10% of their field
0,0%
5,0%
10,0%
15,0%
20,0%
25,0%
30,0%
35,0%
40,0%
A (0 >MNJS <=0.40)
B (0.40 > MNJS <= 0.80)
C (0.80 > MNJS <= 1.20)
D (1.20 > MNJS <=1.60)
E (MNJS > 1.60)
Problems with H-index• Bibliometric-mathematical issues
– mathematically inconsistent (Waltman & van Eck, 2012)
– conservative
– Not field normalized (van Leeuwen, 2008)
• Bibliometric-methodological issues– How to define an author?
– In which bibliographic/metric environment?
• Conceptual/general issues– Favors age, experience, and high productivity (Costas & Bordons, 2006)
– No relationship with research quality
– Ignores other elements of scholarly activity
– Promotes one-indicator thinking
The problem of fields and h-index …• Spinoza candidates, across all domains …
• Use output, normalized impact, and h-index
Soc
HumMat
Soc
Eng
Psy
Eng ChePsyMed
Med
Che
Med
Med
Phy
PhyBio
BioPhy
Psy
Env
Phy
Med
Bio
MedMed
0.00
1.00
2.00
3.00
4.00
5.00
6.00
7.00
0 50 100 150 200 250
TOTAL PUBLICATIONS
CPP
/FC
Sm
Med
Med
Bio
MedPhy Env
PsyPhy
BioBioPhy
Phy MedMed
CheMedMed PsyChe
EngPsyEngSocMatHum
Soc
0
10
20
30
40
50
60
0 50 100 150 200 250
TOTAL PUBLICATIONS
H-in
dex
14
Scientific integrity and misconduct
Scientific integrity and misconduct
• An often mentioned cause for any kind of misconduct is the “Publish or Perish” culture.
• As such, that is supposed to force individuals to produce more and more output, ….
• … as these are considered the building blocks of an academic career !
• However, pressure is on all of us !
Effects of the “Publish or perish” culture: the case of J.H. Schön
0
200
400
600
800
1000
1200
1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012
JH Schon
Effects of the “Publish or perish” culture: the case of D. Stapel
0
50
100
150
200
250
300
350
1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013
Impact Stapel DA
D. Stapel caused the Netherlands to be the country ranked first in the field of Social psychology, when it comes to retractions !
18
Applicability of bibliometrics
Output of a university …
Output, and the way it is represented in Web of Science!
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
(Bio)medicine
Economics & Management
Humanities
Law
Social sciences
All Publications
WoS Publications
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
(Bio)medicine
Economics & management
Humanities
Law
Social sciences
BOOK
CASE
CHAP
CONF
GEN
JOUR
MGZN
PAT
RPRT
THES
Output, and the wide variety not covered in Web of Science !
20
Take home messages
Take-home messages on journals
• Journals tend to publish positive/confirming results.• Editorial boards are driven by market shares as well !• Therefore, selection is harsh, and rejection rates are high
Take-home messages on data• Data are not frequently published.• Therefore, they do not give any credits for the producers!• This keeps most work non transparent• Database with negative results is necessary
Take-home messages on bibliometrics• Ask yourself the question “What do I want to measure ? ”• And also “Can that be measured ? “ • Take care of proper data collection procedures.• Then, always use actual and expected citation data.• Apply various normalization procedures (field, document, age)
• Always have a variety of indicators.• Always include various elements of scholarly activity.
• And perhaps most important, include peer review in your assessment procedures !!!
24
Appendix slides
25
Development of authorship across all domains of scholarly activity
1981
1983
1985
1987
1989
1991
1993
1995
1997
1999
2001
2003
2005
2007
2009
2011
1.00
2.00
3.00
4.00
5.00
6.00
7.00 MULTIDISCIPLINARY JOURNALS
BASIC MEDICAL SCIENCES
BASIC LIFE SCIENCES
BIOMEDICAL SCIENCES
CLINICAL MEDICINE
ASTRONOMY AND ASTROPHYSICS
AGRICULTURE AND FOOD SCIENCE
CHEMISTRY AND CHEMICAL ENGINEERING
BIOLOGICAL SCIENCES
INSTRUMENTS AND INSTRUMENTATION
PHYSICS AND MATERIALS SCIENCE
ENERGY SCIENCE AND TECHNOLOGY
EARTH SCIENCES AND TECHNOLOGY
HEALTH SCIENCES
ENVIRONMENTAL SCIENCES AND TECHNOLOGY
ELECTRICAL ENGINEERING AND TELECOMMUNICATION
PSYCHOLOGY
MECHANICAL ENGINEERING AND AEROSPACE
COMPUTER SCIENCES
CIVIL ENGINEERING AND CONSTRUCTION
GENERAL AND INDUSTRIAL ENGINEERING
EDUCATIONAL SCIENCES
STATISTICAL SCIENCES
SOCIAL AND BEHAVIORAL SCIENCES, INTERDISCI-PLINARY
MANAGEMENT AND PLANNING
SOCIOLOGY AND ANTHROPOLOGY
INFORMATION AND COMMUNICATION SCIENCES
MATHEMATICS
LAW AND CRIMINOLOGY
ECONOMICS AND BUSINESS
LANGUAGE AND LINGUISTICS
POLITICAL SCIENCE AND PUBLIC ADMINISTRATION
HISTORY, PHILOSOPHY AND RELIGION
CREATIVE ARTS, CULTURE AND MUSIC
LITERATURE
26
Coverage issues
27
AU Moed, HF; Garfield, E. in WO
S
TI In basic science the percentage of 'authoritative' references decreases as bibliographies become shorter
SO SCIENTOMETRICS 60 (3): 295-303, 2004 Y
RF ABT HA, J AM SOC INF SCI T, v 53, p 1106, 2004 Y
GARFIELD, E. CITATION INDEXING, 1979 (BOOK!) N
GARFIELD E, ESSAYS INFORMATION S, v 8, p 403, 1985 N
GILBERT GN, SOC STUDIES SCI, v 7, p 113, 1977 Y
MERTON RK, ISIS, v 79, p 606, 1988 Y
ROUSSEAU R, SCIENTOMETRICS, v 43, p 63, 1998 Y
ZUCKERMAN H, SCIENTOMETRICS, v 12, p 329, 1987 Y
WoS Coverage = 5/7 = 71%
Not in WoS
WoS Coverage in 2010 across disciplines
• Black=Excellent coverage (>80%)
• Blue= Good coverage (between 60-80%)
• Green= Moderate coverage (but above 50%)
• Orange= Moderate coverage (below 50%, but above 40%)
• Red= Poor coverage (highly problematic, below 40%)
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
BASIC LIFE SCIENCES (99,991)
BIOMEDICAL SCIENCES (105,156)
MULTIDISCIPLINARY JOURNALS (8,999)
CHEMISTRY AND CHEMICAL ENGINEERING (118,141)
CLINICAL MEDICINE (224,983)
ASTRONOMY AND ASTROPHYSICS (12,932)
PHYSICS AND MATERIALS SCIENCE (137,522)
BASIC MEDICAL SCIENCES (18,450)
BIOLOGICAL SCIENCES (60,506)
AGRICULTURE AND FOOD SCIENCE (26,709)
INSTRUMENTS AND INSTRUMENTATION (8,485)
EARTH SCIENCES AND TECHNOLOGY (33,160)
PSYCHOLOGY (24,244)
ENVIRONMENTAL SCIENCES AND TECHNOLOGY (42,705)
MECHANICAL ENGINEERING AND AEROSPACE (20,336)
HEALTH SCIENCES (29,213)
ENERGY SCIENCE AND TECHNOLOGY (15,021)
MATHEMATICS (27,873)
STATISTICAL SCIENCES (11,263)
GENERAL AND INDUSTRIAL ENGINEERING (8,756)
CIVIL ENGINEERING AND CONSTRUCTION (8,430)
ECONOMICS AND BUSINESS (16,243)
ELECTRICAL ENGINEERING AND TELECOMMUNICATION (...
MANAGEMENT AND PLANNING (7,201)
COMPUTER SCIENCES (23,687)
EDUCATIONAL SCIENCES (9,917)
INFORMATION AND COMMUNICATION SCIENCES (4,006)
SOCIAL AND BEHAVIORAL SCIENCES, INTERDISCIPLINARY...
SOCIOLOGY AND ANTHROPOLOGY (9,907)
LAW AND CRIMINOLOGY (5,299)
LANGUAGE AND LINGUISTICS (3,514)
POLITICAL SCIENCE AND PUBLIC ADMINISTRATION (6,423)
HISTORY, PHILOSOPHY AND RELIGION (11,753)
CREATIVE ARTS, CULTURE AND MUSIC (6,147)
LITERATURE (4,786)
Discipline(Publications in 2010)
% Coverage of references in WoS
Some clear ‘perversions’ of the system … ?
• “You call me, I call you”
• When time is passing by …
• Salami slicing to boost an academic career
• Multiple authorship (without serious contributing)
• Putting your name on everything your unit produces
• The role of self citations
• Jumping on hypes and fashionable issues