THE FUTURE OF BIBLIOMETRICS UNDER THE MICROSCOPE

Philip Purnell SCONUL, Southampton – July, 2015

BACKGROUND & REF

INTELLIGENT INFORMATION FOR PROFESSIONALS 3

THE POWER OF CITED REFERENCES

2003 2007

2011

2008

1965 1998

2001

1930

Times Cited

Related Records

1986

2008 1984

1972

Cited References

“ … association of ideas …”

2002

CITATION BEHAVIOUR VARIES BETWEEN FIELDS

Where did all this come from? • 1955

– Eugene Garfield’s Science paper on “Citation Indexes for Science”

• 1963 – Science Citation Index (ISI >Thomson >Thomson Reuters)

• 1972 – U.S. National Science Foundation initiates Science Indicators (later

Science & Engineering Indicators), including publication and citation data

• 1980s – Uptake of science indicators in Europe; research by SPRU, CWTS,

Hungarian Academy, as well as ISI

• 1992 – Advisory Board for the Research Councils works with ISI on

National Science Indicators to benchmark UK

• 2004 – Elsevier’s Scopus and Google Scholar are launched

7

Where did research evaluation come from? • 1960

– “The white heat of the technological revolution”, Harold Wilson

• 1970 – “For the scientists, the party is over”, Shirley Williams

• 1980 – UGC/ABRC consensus on selectivity – 1986, Research Selectivity Exercise – 1989, Research Assessment Exercise

• 1990 – Evolution of research management and administration – 1992-2008, RAE - the standard model, evolving grades – 2014, Research Excellence Framework

8

We find bibliometric impact and peer review are coherent across institutions (chemistry data)

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

0 1 2 3 4 5 6

Aver

age

cita

tion

impa

ct fo

r uni

vers

ity 1

996-

2000

Relative citation impact of articles submitted for RAE2001

Grade 5*

Grade 5

Grade 4

Grade 3a

Grade 3b

Spearman r = 0.57, P<0.001

Ratio mapped/NSI = 1.93

Data and analysis: Evidence Thomson Reuters

Responses to evaluation Research trajectory changed from mid-1980s

1.1

1.2

1.3

1.4

1.5

1981 1985 1989 1993 1997 2001 2005 2009

Rela

tive

impa

ct o

f UK

rese

arch

pub

licat

ions

UK citation impact

5 yr moving av'ge

Arrows indicate RAE years, e.g. 2001 and 2008

Data and analysis: Evidence Thomson Reuters

Responses to evaluation Behavioural games - Goodhart’s Law

RAE1996 Science Engineering Social sciences Humanities and arts

Outputs % Outputs % Outputs % Outputs % Books and chapters 5,013 5.8 2,405 8.1 16,185 35.1 22,635 44.4

Conference proceedings 2,657 3.1 9,117 30.8 3,202 6.9 2,133 4.2

Journal articles 77,037 89.8 16,951 57.3 22,575 49.0 15,135 29.7

Other 1,104 1.3 1,122 3.8 4,154 9.0 11,128 21.8

RAE2001

Books and chapters 1,953 2.5 1,438 5.4 12,972 28.6 25,217 46.5

Conference proceedings 751 0.9 3,944 14.9 857 1.9 1,619 3.0

Journal articles 76,182 95.8 20,657 78.1 29,449 65.0 17,074 31.5

Other 618 0.8 408 1.5 2,008 4.4 10,345 19.1

RAE2008

Books and chapters 1,048 1.2 216 1.2 12,632 19.0 21,579 47.6

Conference proceedings 2,164 2.5 326 1.8 614 0.9 897 2.0

Journal articles 80,203 93.8 17,451 95.4 50,163 75.5 14,543 32.1

Other 2,125 2.5 301 1.6 3,018 4.5 8,287 18.3

Data and analysis: Evidence Thomson Reuters

12

Research policy and management is about ‘more, better research’

Research quality

Research black box

What we want to know

Output data have underpinned quantitative research evaluation

What we have to use

Research quality

Research black box

O U T P U T S

Journals and proceedings Citations

What we want to know

You can now use comprehensive research management information

Research quality

Research black box

Numbers – of researchers,

facilities, collaboration

O U T P U T S

Journals, books &

proceedings

IDEAS proposals,

applications and

partnerships

Trained people

Licences and spin outs

Patents

Deals and revenue

Citation and address

links

Skilled employment

Industrial contracts

Charitable awards

O U T C O M E S

Reports and grey literature

Citations

Social policy change

What evaluators want to know

What evaluation needs to use

Research scholarships

Innovation funds

Research grants

I N P U T S

What research users want to know

Data and analysis: Evidence Thomson Reuters These data, added to peer review, create a modern ‘gold standard’

They are proxy indicators, not metrics We use multiple ‘bearings’ to assess our uncertainty

Data and analysis: Evidence Thomson Reuters

CONTROVERSY, LIMITATIONS & TRENDS Article level indicators Rankings vs profiles Social impact vs citation impact Visualisation Scientific approach Balance with peer evaluation

17

CONTROVERSY

18

JOURNAL EDITORIAL POLICY SELECTION CRITERIA

40% of the journals:

• 80% of the publications

• 92% of cited papers

4% of the journals:

• 30% of the publications

• 51% of cited papers

0

20

40

60

80

100

120

0 1000 2000 3000 4000 5000 6000

# of journals

% o

f dat

abas

e

Articles Citations

… exponentially diminishing returns of extending a search for references in science journals … Samuel C Bradford 1934

LIMITATIONS TO BIBLIOMETRICS

Source data • Representation

Outputs • Papers, patents, products, procedures, spin-offs

Impact • Motivation for citation • Citation, Social, Economic

WHY DO PEOPLE CITE PAPERS?

List of reasons:

Implications:

LIMITATIONS TO BIBLIOMETRICS NEGATIVE CITATIONS

LIMITATIONS TO BIBLIOMETRICS SELF-CITATIONS

• REV BRAS FARMACOGN: First Journal Impact Factor in 2009=3.462

Journals whose rank in category is significantly distorted by self-citation are removed from JCR for 2 years, then re-evaluated

LIMITATIONS TO BIBLIOMETRICS MUTUAL CITATIONS

24

LIMITATIONS TO BIBLIOMETRICS MULTI-AUTHORED PAPERS

IMPACT OF A PAPER

26

BENCHMARKING A PAPER

27

IMPACT OF A PAPER

28

24 / 13.82 = 1.74 times journal expected citations

NORMALISATION ARTICLE LEVEL INDICATORS

29 Source: Thomson Reuters InCites

This paper has received 45/16.74 = 2.69 times the expected citations for this subject category

This paper has received 45/21.48 = 2.09 times the expected citations for this journal

Integrative & Complementary Medicine articles from 2005 have been cited 16.74 times

Articles published in this journal from 2005 have been cited 21.48 times

EVALUATING INDIVIDUAL RESEARCHERS

Computer Science

Medicine

METHODS & INDICATORS UNIVERSITY RANKINGS

METHODS & INDICATORS ALT METRICS

Alternative impacts (other than traditional citation counting) Usage/Views: Downloads Discussed: Journals, blogs, Twitter, Wiki, Book reviews Saved: mendeley, Cite U Like Citations: Web of Science Recommendations: F1000Prime Several publishers have started providing data BioMed, PLOS, NPG, Elsevier. BMJ study Funders showing interest UK MRC Limitations size & selectivity of database (stat significance) Standardisation of data Validity of indicators

MAPPING / VISUALISING SCIENCE

The Future of Bibliometrics - Summary • Here to stay

• Increasingly scientific – both data and evaluation

• Include all research outputs

• Social impact - Altmetrics

• Backward chaining from outcomes

• Visualisations

• Library to play leading role

• Maintain balance with peer review

34