The role of bibliometrics in

research management ...

It’s complicated !

Dr. Thed van Leeuwen

Center for Science & Technology Studies (CWTS)

TU Delft seminar on “Are you ready to publish ?” , February 5th

2015

Outline

• Bibliometrics and research management context

• Infamous bibliometric indicators

• Scientific integrity and misconduct

• Applicability of bibliometrics

• Take-home messages …

1

Bibliometrics and

the research

management

context

2

What is bibliometrics ?

• Bibliometrics can be defined as the quantitative analysis of science and technology (development), and the study of cognitive and organizational structures in science and technology.

• Basic for these analyses is the scientific communication between scientists through (mainly) journal publications.

• Key concepts in bibliometrics are output and impact, as measured through publications and citations.

• Important starting point in bibliometrics: scientists express, through citations in their scientific publications, a certain degree of influence of others on their own work.

• By large scale quantification, citations indicate (inter)national influence or (inter)national visibility of scientific activity, but should not be interpreted as synonym for ‘quality’.

‘Classical’ image of the Credibility cycle

Credibility cycle (adapted from Latour and Woolgar (1979) & Rip (1990)

PEER REVIEW

Rise of performance indicators & bibliometrics

Need for formalised measures

• ‘Push’ from science policy (from 1970s onwards)

• Independent of peer review

• New Public Management / Neo-liberalism (from 1980s onwards)

Matrix-structure science (Whitley)

• Researchers part of international community (Peer review)

• But also part of local institutions (Specific management practices, e.g. appraisals, external evaluations)

• Institute managers not always part of international expert community

• Tighter forms of management (from the 1990s onwards)

Distance

Extended credibility cycle

Infamous

bibliometric

indicators:

JIF & H-index

7

Definitions of Journal Impact Factor & Hirsch Index

• Definition of JIF:

– The mean citation score of a journal, determined by dividing all

citations in year T by all citable documents in years T-1 and T-2.

• Definition of h-index:

– The ‘impact’ of a researcher, determined by the number of received

citations of an oeuvre, sorted by descending order, where the

number of received citations on that single paper equals the rank

position.

Problems with JIF

• Methodological issues

– Was/is calculated erroneously (Moed & van Leeuwen, 1996)

– Not field normalized

– Not document type normalized

– Underlying citation distributions are highly skewed (Seglen, 1994)

• Conceptual/general issues

– Inflation (van Leeuwen & Moed, 2002)

– Availability promotes journal publishing

– Is based on expected values only

– Stimulates one-indicator thinking

– Ignores other scholarly virtues

Deconstructing the myth of the JIF…

• Take the Dutch output

• Similar journal impact classes

• Focus on publications that belong to the top 10% of their field

0,0%

5,0%

10,0%

15,0%

20,0%

25,0%

30,0%

35,0%

40,0%

A (0 >MNJS <=0.40)

B (0.40 > MNJS <= 0.80)

C (0.80 > MNJS <= 1.20)

D (1.20 > MNJS <=1.60)

E (MNJS > 1.60)

Problems with H-index

• Bibliometric-mathematical issues

– mathematically inconsistent (Waltman & van Eck, 2012)

– conservative

– Not field normalized (van Leeuwen, 2008)

• Bibliometric-methodological issues

– How to define an author?

– In which bibliographic/metric environment?

• Conceptual/general issues

– Favors age, experience, and high productivity (Costas & Bordons, 2006)

– No relationship with research quality

– Ignores other elements of scholarly activity

– Promotes one-indicator thinking

The problem of fields and h-index …

• Spinoza candidates, across all domains …

• Use output, normalized impact, and h-index

Soc

HumMat

Soc

Eng

Psy

Eng ChePsyMed

Med

Che

Med

Med

Phy

PhyBio

Bio

Phy

Psy

Env

Phy

Med

Bio

MedMed

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

0 50 100 150 200 250

TOTAL PUBLICATIONS

CP

P/F

CS

m

Med

Med

Bio

MedPhy Env

PsyPhy

BioBioPhy

Phy Med

MedCheMed

Med Psy

Che

EngPsy

EngSoc

MatHum

Soc

0

10

20

30

40

50

60

0 50 100 150 200 250

TOTAL PUBLICATIONS

H-i

nd

ex

In what database context …

Database H-index Based upon …

Web of Science 14 Articles in journals

Scopus 25 Articles, book (chapters), and

conference proceedings papers

Google Scholar 32 All types, incl. Reports

13

Selected my own publications in WoS and Scopus, Google Scholar

has a pre-set profile.

Scientific integrity

and misconduct

14

Scientific integrity and misconduct

• An often mentioned cause for any kind of

misconduct is the “Publish or Perish” culture.

• As such, that is supposed to force individuals to

produce more and more output, ….

• … as these are considered the building blocks of an

academic career !

• However, pressure is on all of us !

Effects of the “Publish or perish”

culture: the case of J.H. Schön

0

200

400

600

800

1000

1200

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

JH Schon

Effects of the “Publish or perish”

culture: the case of D. Stapel

0

50

100

150

200

250

300

350

1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Impact Stapel DA

D. Stapel caused the Netherlands to be the country ranked first

in the field of Social psychology, when it comes to retractions !

Applicability of

bibliometrics

18

Output of a university …

Output, and the way it is

represented in Web of Science!

0% 10% 20% 30% 40% 50% 60% 70% 80% 90%

(Bio)medicine

Economics & Management

Humanities

Law

Social sciences

All Publications

WoS Publications

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

(Bio)medicine

Economics & management

Humanities

Law

Social sciences

BOOK

CASE

CHAP

CONF

GEN

JOUR

MGZN

PAT

RPRT

THES

Output, and the wide variety

not covered in Web of Science !

Take home

messages

20

Take-home messages on journals

• Journals tend to publish positive/confirming results.

• Editorial boards are driven by market shares as well !

• Therefore, selection is harsh, and rejection rates are high

Take-home messages on data

• Data are not frequently published.

• Therefore, they do not give any credits for the producers!

• This keeps most scientific work non transparent

• Databases with negative results are necessary

Take-home messages on bibliometrics

• Ask yourself the question “What do I want to measure ? ”

• And also “Can that be measured ? “

• Take care of proper data collection procedures.

• Then, always use actual and expected citation data.

• Apply various normalization procedures (field, document, age)

• Always have a variety of indicators.

• Always include various elements of scholarly activity.

• And perhaps most important, include peer review in

your assessment procedures !!!

Thank you for your attention!

Any questions?

Ask me, or mail me

leeuwen@cwts.nl

23

Appendix slides

24

Development of authorship across all

domains of scholarly activity

25

1,00

2,00

3,00

4,00

5,00

6,00

7,00 MULTIDISCIPLINARY JOURNALS

BASIC MEDICAL SCIENCES

BASIC LIFE SCIENCES

BIOMEDICAL SCIENCES

CLINICAL MEDICINE

ASTRONOMY AND ASTROPHYSICS

AGRICULTURE AND FOOD SCIENCE

CHEMISTRY AND CHEMICAL ENGINEERING

BIOLOGICAL SCIENCES

INSTRUMENTS AND INSTRUMENTATION

PHYSICS AND MATERIALS SCIENCE

ENERGY SCIENCE AND TECHNOLOGY

EARTH SCIENCES AND TECHNOLOGY

HEALTH SCIENCES

ENVIRONMENTAL SCIENCES AND TECHNOLOGY

ELECTRICAL ENGINEERING ANDTELECOMMUNICATIONPSYCHOLOGY

MECHANICAL ENGINEERING AND AEROSPACE

COMPUTER SCIENCES

CIVIL ENGINEERING AND CONSTRUCTION

GENERAL AND INDUSTRIAL ENGINEERING

EDUCATIONAL SCIENCES

STATISTICAL SCIENCES

SOCIAL AND BEHAVIORAL SCIENCES,INTERDISCIPLINARYMANAGEMENT AND PLANNING

SOCIOLOGY AND ANTHROPOLOGY

INFORMATION AND COMMUNICATION SCIENCES

MATHEMATICS

LAW AND CRIMINOLOGY

ECONOMICS AND BUSINESS

LANGUAGE AND LINGUISTICS

POLITICAL SCIENCE AND PUBLIC ADMINISTRATION

HISTORY, PHILOSOPHY AND RELIGION

CREATIVE ARTS, CULTURE AND MUSIC

LITERATURE

Coverage issues

26

27

AU Moed, HF; Garfield, E. in

W

O

S

TI In basic science the percentage of 'authoritative' references

decreases as bibliographies become shorter

SO SCIENTOMETRICS 60 (3): 295-303, 2004 Y

RF ABT HA, J AM SOC INF SCI T, v 53, p 1106, 2004 Y

GARFIELD, E. CITATION INDEXING, 1979 (BOOK!) N

GARFIELD E, ESSAYS INFORMATION S, v 8, p 403, 1985 N

GILBERT GN, SOC STUDIES SCI, v 7, p 113, 1977 Y

MERTON RK, ISIS, v 79, p 606, 1988 Y

ROUSSEAU R, SCIENTOMETRICS, v 43, p 63, 1998 Y

ZUCKERMAN H, SCIENTOMETRICS, v 12, p 329, 1987 Y

WoS Coverage

= 5/7 = 71%

Not in WoS

WoS Coverage in 2010 across disciplines

• Black=Excellent coverage (>80%)

• Blue= Good coverage (between 60-80%)

• Green= Moderate coverage (but above 50%)

• Orange= Moderate coverage (below 50%, but above 40%)

• Red= Poor coverage (highly problematic, below 40%)

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100

%

BASIC LIFE SCIENCES (99,991)

BIOMEDICAL SCIENCES (105,156)

MULTIDISCIPLINARY JOURNALS (8,999)

CHEMISTRY AND CHEMICAL ENGINEERING (118,141)

CLINICAL MEDICINE (224,983)

ASTRONOMY AND ASTROPHYSICS (12,932)

PHYSICS AND MATERIALS SCIENCE (137,522)

BASIC MEDICAL SCIENCES (18,450)

BIOLOGICAL SCIENCES (60,506)

AGRICULTURE AND FOOD SCIENCE (26,709)

INSTRUMENTS AND INSTRUMENTATION (8,485)

EARTH SCIENCES AND TECHNOLOGY (33,160)

PSYCHOLOGY (24,244)

ENVIRONMENTAL SCIENCES AND TECHNOLOGY (42,705)

MECHANICAL ENGINEERING AND AEROSPACE (20,336)

HEALTH SCIENCES (29,213)

ENERGY SCIENCE AND TECHNOLOGY (15,021)

MATHEMATICS (27,873)

STATISTICAL SCIENCES (11,263)

GENERAL AND INDUSTRIAL ENGINEERING (8,756)

CIVIL ENGINEERING AND CONSTRUCTION (8,430)

ECONOMICS AND BUSINESS (16,243)

ELECTRICAL ENGINEERING AND TELECOMMUNICATION (...

MANAGEMENT AND PLANNING (7,201)

COMPUTER SCIENCES (23,687)

EDUCATIONAL SCIENCES (9,917)

INFORMATION AND COMMUNICATION SCIENCES (4,006)

SOCIAL AND BEHAVIORAL SCIENCES, INTERDISCIPLINARY...

SOCIOLOGY AND ANTHROPOLOGY (9,907)

LAW AND CRIMINOLOGY (5,299)

LANGUAGE AND LINGUISTICS (3,514)

POLITICAL SCIENCE AND PUBLIC ADMINISTRATION (6,423)

HISTORY, PHILOSOPHY AND RELIGION (11,753)

CREATIVE ARTS, CULTURE AND MUSIC (6,147)

LITERATURE (4,786)

Discipline

(Publications in 2010)

% Coverage of references in WoS

Some clear ‘perversions’ of the system … ?

• “You call me, I call you”

• When time is passing by …

• Salami slicing to boost an academic career

• Multiple authorship (without serious contributing)

• Putting your name on everything your unit produces

• The role of self citations

• Jumping on hypes and fashionable issues