Measuring research impact with bibliometrics

Friday 30th JanuaryTanya Williamson, Assistant Librarian

Overview

Learn about the most common bibliometrics and the tools used to measure research impact and influence.

1. What are bibliometrics?

2. Journal metrics

3. Personal metrics

4. Article level metrics and altmetrics

Optional practical: Have a go at finding a journal's impact factor, and calculating a researcher’s h-index.

• Practical – Find journal metrics using journal citation reports

• Practical – Find a researcher’s h-index using Google Scholar or Web of Science

• Bibliometrics are quantitative methods of analysing research publications and citations

• Based on the assumption that citations = impact

What are bibliometrics?

Citation counts in Web of Science

Visualisation of citation networks from InCites Journal Citation Reports

Graphic from Eigenfactor.org

An attempt to compare the influence and impact of journal titles in a particular discipline. Can be used to inform where to publish, which could in turn improve research impact.

Commonly used metrics available through Journal Citation Reports, part of Web of Science

• Journal Impact Factor

• 5 year Journal Impact Factor

• Immediacy index

• Eigenfactor score

and

• SCImago Journal rank

Journal metrics

Journal Citation Reports category rank

An example: Journal of Allergy and Clinical Immunology

ALLERGY category rank

Journal Impact Factor

JIF

2011 2012 2013 2014

Citations to 2011 publications

Citations to 2012 publications

The JCR year

Items published

Items published

2015

175 125+

250 350+

= 2

The Journal Impact Factor is the average number of times articles from the journal published in the past two years have been cited in the JCR year.

The Journal of Important Discoveries Other Journals in WoS

The Journal Impact Factor is the average number of times articles from the journal published in the past two years have been cited in the JCR year.

Also available: 5 year Journal Impact Factor

Journal Impact FactorAn example: Journal of Allergy and Clinical Immunology

• The Immediacy Index is the average number of times an article is cited in the year it is published.

• Good for comparing cutting edge research field

Immediacy IndexAn example: Journal of Allergy and Clinical Immunology

• Based on source data from Web Of Science

• Not influenced by journal self-citation

Eigenfactor and Article InfluenceAn example: Journal of Allergy and Clinical Immunology

Demo of Journal Citation Reports

SCImago journal rankAn example: Journal of Allergy and Clinical Immunology

• SCImago Journal Rank uses source data from Scopus

• Based on the Google PageRank algorithm

• Weights citations from high prestige journals and attempts to balance the influence of the size of a journal

Immunology and Allergy category rank

• Don’t work well for every discipline due to different publishing patterns and coverage of different publication types

• Mustn’t be used to compare different disciplines

Limitations of journal metrics

Top scorer in Premier league is Luis Suarez with 31 goals

Top scorer in NBA is Kevin Durant with 849 goals

Can we conclude that Durant is a better sportsman?

• Focusing only on Journal Impact Factors could lead to you overlooking smaller, specialist publications

• Dependent on the coverage in the Web of Science database

• Can we infer individual impact from journal impact?

Limitations of journal metrics

This video: Bibliometrics for the Individual, Prof Dermot Diamond

Measure your Research Impact, Bibliometrics toolkit on YouTube

Bibliometrics for individuals

Key metrics

• Citation counts - uses a particular dataset to count how many citations an author has received

• H-index, devised by: J. E. Hirsch in 2005

• There are several ‘improvements’ on the h-index, e.g. the g-index, the e-index, the w-index, the hfa index…

Personal metrics

Easy explanation:

If a scholar has 20 articles which have each been cited 20 times, s/he has an h-index of 20

• Web of Science

Personal metrics

Demo of Web of Science

Google Scholar

• A way to find your own h-index

• The author needs to have a Google Scholar profile

Personal metrics

Publish or Perish

• Uses Google Scholar data

• Author and journal metrics

• Accompanying book and website

• Name ambiguity – getting a comprehensive list is not easy, even in Web of Science

• No one metric can capture all citations or ‘impacts’

• Early career researchers will be at a disadvantage

• Citations ≠ endorsements of quality!

Limitations of personalmetrics

Resistance against the h-index, ImpactStory

• Citation analysis in Web of Science, Google Scholar

• Publishers’ interfaces often include citing articles

Article level metrics

Publisher’s site

Google ScholarWeb of Science

“Article-Level Metrics (ALMs) leverage the acceleration of research communication made possible by the networked landscape of researcher tools and services. Also by incorporating the manifold ways in which research is disseminated, these article impact indicators are made available rapidly after publication and are continually updated.”

PLOS One Article Level Metrics Information

Alternative metrics or altmetrics

ImpactStory: like a living publications CV. Adding metrics all the time https://impactstory.org/metrics

Altmetric.com: watch social media sites, newspapers, government policy documents and other sources for mentions of scholarly articles to compile article level metrics.

• Shows the broader, societal impact

• More than just articles

Remember that highly cited article from before?

Limitations of article level and altmetrics

Source of data Citation to an article

Web of Science 2021

Google Scholar 2937

Publisher’s website 1478

• Publish or Perish http://www.harzing.com/pop.htm

• Measure Your Research Impact toolkit http://www.ndlr.ie/myri/

• Conference Rankings for Computer Science: http://www.core.edu.au/index.php/conference-rankings

• The Ultimate Guide to Altmetrics (Researchers edition) http://impactstory.demo.libguides.com/c.php?g=211074

The University is currently considering other tools, such as Scopus, SciVal and Altmetric for Institutions.

Thanks to Scott Taylor from University of Manchester for permitting me to borrow the sporting analogy from his presentation with Stephen Pearson at the Northern Collaboration Conference 2014. http://www.slideshare.net/northerncollaboration/northern-collaboration-conference-2014-40398613

Follow up resources

1. Navigate to Journal Citation Reports from the Library Databases A-Z

http://www.lancaster.ac.uk/library

2. Choose a subject category to compare journals against different metrics

Practical

Find a scholar’s H-index in Google Scholar

1. Go to Google Scholar http://scholar.google.co.uk/

2. Search for a topic of your interest

3. Choose a scholar who has a Google Scholar profile

…

Outside of the session, you can look into Publish or Perish software available from http://www.harzing.com/pop.htm which ingests data from Google Scholar and allows you to analyse on a variety of journal and personal metrics.

Practical