the strange world of bibliometric numbers: implications for professional practice

The strange world of bibliometric numbers: Implications for professional practice

Dr Ian RowlandsDavid Wilson Library

Manchester Metropolitan University27 June 2016

Three themes

Look at the underlying data

• don’t take indicators at face value

Think how your data will be used

• put your numbers in context

Accept that bibliometrics is driven by rare events

• put measurements around that uncertainty

Content

The importance of context: Interpreting the h-index

The journal impact factor: A case study in extremes

Working with `difficult’ numbers

PrefaceBibliometrics deals with rare events

Fly body length (mm)

Fly body length (mm)Statistic ValueMean 45.5

Median 45.5

Mode 45

Range 36 – 55

Standard deviation 3.9

Citation frequencies: Nature 2008

Citations to present for 975 Nature articles and review papers published in 2008

Nature citationsStatistic ValueMean cites per paper 275.1

Median 164

Mode 1

Range 0 – 4,735

Standard deviation 366.6

Citations to present for 975 Nature articles and review papers published in 2008

Nature citationsStatistic ValueMean cites per paper 275.1

Median 164

Mode 1

Range 0 – 4,735

Standard deviation 366.6

Citations to present for 975 Nature articles and review papers published in 2008

What’s the average??

The data range over three orders of magnitude!!

A thought experiment

What if flies’ body lengths followed the same distribution as citations?

• most typically, a fly would not even exist (often, mode=0)

• 85 per cent of flies would have bodies shorter than average for the whole population, and most would be hors d’oeuvres for the top 15 per cent

• some flies would measure a giant 30-inches

Lesson

`Average’ is a problematic concept in bibliometrics

This has serious implications for

• methodology

• interpretation

• application

The importance of contextInterpreting the h-index

What is the h-index?

What is the h-index?

36

Interpreting the h-index

Harry– 60 papers– 6,000 citations– 100 citations per paper

Tom– 60 papers– 6,000 citations– 100 citations per paper

Interpreting the h-index

Harry– 60 papers– 6,000 citations– 100 citations per paper

Tom– 60 papers– 6,000 citations– 100 citations per paper

h-index = 20

Interpreting the h-index

Harry– 60 papers– 6,000 citations– 100 citations per paper

Tom– 60 papers– 6,000 citations– 100 citations per paper

h-index = 20

h-index = 40

The h-index measures consistency not absolute impact.

Quite a few Nobel laureates have low to moderate h-indexes …

On the h-index and its variants

“These are often breathtakingly naïve attempts to capture a complex citation record with a single number. Indeed the primary advantage of these new indices over simple histograms of citation counts is that the indices discard almost all of the detail … and this makes it possible to rank any two scientists … Surely understanding ought to be the goal when assessing research, not ensuring that any two people are comparable.”

International Mathematical Union, Citation Statistics, June 2008, p.14http://www.mathunion.org/fileadmin/IMU/Report/CitationStatistics.pdf

Practical tips

The accuracy of h depends on not missing any relevant papers in the core as well as avoiding false drops

Present h with a health warning that pushes responsibility for curating their online identity back on the client (e.g. ORCID, active management of their ResearcherID)

Source coverage (particularly Scopus vs Web of Science) is a seriously overlooked issue and may yield very different h values

Since h throws away information about important highly cited papers (papers with citations > h) it does many researchers a disservice

The journal impact factorA case study in extremes

Dr Eugene Garfield

Journal impact factor 2015 calculation

citations accruedduring 2015

papers published in 2013

papers published in 2014

+

÷

Numerator=ALL citations

Denominator=articles and reviews only

Bibliometric ratios can be very unstable

The journal impact factor is a simple ratio:

JIF = citations / papers

Citations can throw up surprises, and these will be amplified if the sample is small.

Journal impact factor instability

0 100 200 300 400 500 600 700 800 900 1000-125%

-100%

-75%

-50%

-25%

0%

25%

50%

75%

100%

125%

150%

175%

Journal size (2011 articles)

Mea

n %

chan

ge in

IF (2

011

on 2

010)

What happened here?

Acta Crystallographica Section ACitations received in

2008 2009 2010

The whole journal 3,628 6,068 7,325

George Sheldrick, A short history of SHELX (2008) 64(1) pp 112-122. 3,542 5,897 7,029

Helen Berman, The Protein Data Bank: A historical perspective (2008) 64(1) pp 88-95. 4 7 23

A short history of SHELX

Abstract

“An account is given of the development of the SHELX system of computer programs from SHELX-76 to the present day …This paper could serve as a general literature citation when one or more of the open-source SHELX programs … are employed in the course of a crystal-structure determination.”

George M Sheldick, A short history of SHELX, Acta Crystallographica Section A (2008) 64(1): 112-122.

top 10% of articles generate

40% of all citations …

… 82% of articles are `below average’Bill Gates gets on the train …

and, on average, everyone on

board is a millionaire

(at least until he gets off)

Lessons

The example of Acta Crystallographica A’s 2009 JIF is a salutary reminder that rare events do happen. The issue is compounded in this case because the denominator is small (127 papers).

How could the journal impact factor (and other bibliometrics indicators) be better presented?• in principle, the mode and median are far more appropriate and

informative than the mean when dealing with highly skewed distributions

• but in reality, the mode and median for many indicators will simply be 0 or 1

• but this is not terribly realistic strategy!

Working with `difficult’ numbersData transformation and stability intervals

Nature 2008 (n=945) cites to end 2015

Nature 2008 (n=945) cites to end 2015

A simple logarithmic data transform

Advantages

By using a logarithmic rather than a linear scale, the mode, median and mean converge and we have a much better sense of the central tendency.

This has three practical benefits:

• Suddenly `average’ becomes meaningful

• You can now use a whole range of statistical tests that assume a normal distribution (e.g. student’s t-test, ANOVA)

• You can now put 95% confidence intervals around the mean, which aids interpretation

Health warning

YOU MUST LOOK AT THE DATA

Fairly mature citation distributions are often approximately loglinear but this is not always the case.

Try other transforms (e.g. square root, reciprocal) to see if they offer a better solution.

If you want to be squeaky clean, consider a Box-Cox test to find the optimal transform.

Stability intervals

Stability intervals

Final conclusions

Always look at the raw data, not the cooked indicator, and think about context

`Rare events’ can make a huge difference

Bibliometric indicators are unstable and this can lead to poor decision-making

You have a responsibility to present meaningful averages and to put bounds around data uncertainty