Measuring Research Impact

Dr. Neeraj ChaurasiaIndian Institute of Technology (IIT) Delhi

neerajkc@library.iitd.ac.in

Outlines

Research Impact

What we are Measuring ?

Why Assess Scholarly Impact of Research ?

How do we Measure the Impact of our Work?

Researcher Impact

Research Metrics

An Overview of available Metrics and Tools

Altmetrics

Role of Libraries and Librarians

What is Research Impact?

"Research impact is the demonstrable

contribution that excellent research makes to

society and the economy. Impact embraces all

the extremely diverse ways in which research-

related knowledge and skills benefit individuals,

organizations and nations including academic,

economic, and societal impacts...”

(Research Councils UK)

What we are

Measuring ?

Journal Quality

Article reach

Article impact/Influence

Dissemination

Researcher Impact

Why Scholarly Impact of Research need to be Measure ?

Strengthen CV for promotion or tenure

Showing individual or collective productivity

Quantify ROI on research for grant renewals and progress reports

For future funding by illustrating the value of your research

Identify who is using your work and confirm that it is appropriately credited

Identify collaborators within or outside of your discipline

Benchmarking - self, research group, department, or institution

Measuring the Citations

Showcase network of collaborators

How are we Measuring?

quantitative methods – such as - citation counts, the h-index,

and journal impact factors

there is no one tool or system that completely measures impact

of research

each database or tool uses its own measurement systems,

indices, data and authority files

difficult to use these tools to compare across disciplines that

have different research and publication practices.

Research Metrics

➢Quantitative analysis of scientific and scholarly research outputs andtheir impacts.

➢ Include a variety of measures and statistical methods for assessing thequality and broader impact of scientific and scholarly research, as wellas to track researcher impact

➢ Measure impact and provide insight into the influence of specificjournal publications, individual articles, and authors.

Research Metrics - How are we Measuring? –

Journal-Level Metrics

Measure quality of Journal using citation formulas, such as Impact Factor, Helps to track citation patterns within journals anddetermine which journals are highly-cited.

Article-Level MetricsArticle Metrics, or citation tracking, is used to determine if an article, book, journal, or particular author has been cited by another work.

Author-Level Metrics -Measure bibliographic impact of individual authors

-Measure the impact and productivity of a researcher.

AltmetricsMeasures and monitors reach and impact of research through online interactions like soial media mentions, data sets, websites, blog posts, and more…

Some of the more common metrics and tools you can use to measure research impact

Different Level of Metrics

Journal-Level Metrics

Some of common metrics and evaluation tools used to evaluate and measure a journal’s impact

Journal Impact Factor (JIF) CiteScore SCImago Journal

Ranking (SJR)

Source Normalized Impact

per Paper (SNIP)

Measure

of...

Attempt to quantify the

importance of a particular

journal in a field.

Frequency with which the

'average article' in a journal

has been cited in a particular

year or other defined time

period

Calculates the average

number of citations

received in a calendar

year by all items

published in that journal

in the preceding three

years

Measure of prestige;

accounts for number of

citations received by a

journal and importance

of journals that

citations came from

Normalizes for differences in

citation behavior between

subject fields.

Measures citation impact by

weighting citations based on

the total number of citations

in a subject field

Calculation A: Number of citations in the

current year to papers

published in the journal in the

previous two years

B: Total number of articles

published in the journal in the

previous two years

Impact Factor = A/B

Counts all documents

since they all have the

potential to attract

citations,

CiteScore is independent

of the document-type

classification

Citations from more

prestigious journals

(higher SJR) weighted

more than citations

from less prestigious

journals (lower SJR)

Citations from subject fields

in which citations are less

likely are weighted more

Where can

you get it?

Journal Citation Reports

Clarivate Analytics

Find CiteScore at

https://journalmetrics.sc

opus.com/ Elsevier

free from SCImago

Journal & Country Rank

Elsevier

Scopus OR free from CWTS

Journal Indicator

Journal Impact Factor (JIF)

Number of citations to a journal in a given year from articles occurring in

the past 2 years, divided by the number of scholarly articles published in

the journal in the past 2 years

For example, the 2019 impact factor for a journal would be calculated as follows:

A: Number of citations in the current year(2019) to papers published in the journal in the previous two years (2017 & 2018)

B: Total number of articles published in the journal in the previous two years (2017 & 2018)

2019 impact factor = A/B

e.g. 600 citations = 2

150 + 150 articles

Scopus-Based Metrics CiteScore

Calculates the average number of citations received in a calendar year by all items published in that journal in the

preceding 3 years

CiteScore counts all documents since they all have the potential to attract citations, and the Impact Factor counts

the documents considered most likely to attract citations.

CiteScore is independent of the document-type classification

Find CiteScore at https://journalmetrics.scopus.com/

SJR Indictor (SciMago) Measure of prestige;

Citations are weighted, depending on the rank of the citing journal

A citation from an important journal will count as more than one citation; a citation coming from a less important

journal will count as less than one citation.

Find at www.scimagojr.com

SNIP (Source Normalized Impact per Paper) SNIP weighted citations based on the total number of citations in a subject field.

Normalizes for differences in citation behavior between subject fields.

Corrects for subject-specific characteristics of the field someone is publishing in so any two journal can be compared

Find at CWTS Journal Indicators http://www.journalindicators.com/

Author Level Metrics

h-Index and i10-Index

h-Index i10-IndexMeasure : Measure both productivity and impact of publications

Meaningful when compare to others within the same discipline area

Calculation: A scholar with an index of h has published h paperseach of which has been cited by others at least htimes..

e.g. - If a researcher has 20 papers, each of which hasat least 20 citations, their h-index is 20.

e.g. - An h-index of 10 tells us that an author haspublished 10 papers which have each been cited atleast 10 times

It was introduced in 2011 by Google aspart of their work on Google Scholar,

i10 index refers to the number ofpaper with 10 or more citations.

Limitations:Inaccurate measure of early career researcher impact. Only used in Google Scholar

Where to Find h-Index

Value of the h-index may vary depending on the : source of

information, number of indexed publications, time span, etc.

Citation databases

Web of Science

Scopus

Google Scholar

Requires creation of Google Scholar profile before providing metrics

H-index tends to be higher than what is calculated by Web of Science

Definition

A scientist has index h if h of his/her Np papers have at least h

citations each, and the other (Np − h) papers have no more than

h citations each.

h-Index

A scholar with an index of h has published h papers each of which has been cited by others at least h times

20 papers

cited 20 times or more

Calculate Your h-index - Manually

To manually calculate your h-index, organize articles in descending order, based on the number of times they have been cited....

Web of Science, Scopus, and Google Scholar can also be used to calculate an h-index for that particular citation-tracking database.

In the below example, an author has 8 papers that have been cited 33, 30, 20, 15, 7, 6, 5 and 4

times. This tells us that the author's h-index is 6.

What does an h-index of 6 mean?

An h-index of 6 means that this author has published at least 6 papers that have each received at

least 6 citations.

H-index - Interpretation

14-05-2020

14-05-2020

In the below example, an author has 8 papers that have been cited 33, 30, 20, 15, 7, 6, 5 and 4 times. This tells us that the author's h-index is 6.

What does an h-index of 6 mean?•An h-index of 6 means that this author has published at least 6 papers that have each received at least 6 citations.

Interpreting

14-05-2020

In the below example, an author has 8 papers that have been cited 33, 30, 20, 15, 7, 6, 5 and 4 times. This tells us that the author's h-index is 6.

What does an h-index of 6 mean?•An h-index of 6 means that this author has published at least 6 papers that have each received at least 6 citations.

Interpreting

Web of Science (h-Index)

Web of Science (h-Index)

Google Scholar (h-Index)

Google Scholar - Metrics

H-Index Limitations

• h-index increases with age so comparing

productivity of younger researchers is

problematic

• Calculated in controlled databases but need

comprehensive citation report of all author’s

publications

• Different databases yield different h-index

scores

• Can comparing scientists working in the same

field;

• Citation conventions differ widely among

different fields.

Example - h-index of a

author :

Coverage is different :

Scopus

publications indexed = 10

H-index= 3

Google Scholar

publications indexed = 28

H-index = 6

Web of Science

publications indexed = 5

H-index = 1

Article-Level Metrics

Article-Level

Metrics

Citation-based and altmetric measures can show impact of individual research publication

How many times was an article cited

How is it tracking in social media

What is the geographic distribution of citing papers

What is the disciplinary distribution of citing papers

What is the impact outside of the scholarly Community

Altmetrics address

Ref : http://pitt.libguides.com/altmetrics/introduction

How many times something is downloaded

Who is reading the work

Has is it been covered by

news outlets

Who is commenting on

the work

How is it being shared

Which countries are looking at

my work

Altmetrics

Altmetrics

Coined by Jason Priem in 2010

Also known as; Alternative Metrics/ Article Level Metrics (ALM) or ‘Alt-metrics’

Alternative to more traditional citation impact metrics, such as impact factor and h-index

Altmetrics include social media activity, coverage in media outlets, and inclusion in policy documents or scholarly commentary, among other types of activity.

Several publishers are providing such information to readers, including BioMed Central, Public Library of Science, Frontiers, Nature Publishing Group, Elsevier, Research Gate, academic.edu

Some Altmetrics Tools

Altmetrc

ImpactStory

ReaderMeter

ScienceCard

PLoS impact explorer

PLoS article level metrics

PaperCritic

Crowdometer

PlumX

Usage

HTML views, PDF/XML downloads (various sources – eJournals, PubMed Central,

FigShare, Dryad, etc.)

Captures

CiteULike bookmarks, Mendeley readers/groups, Delicio.us

Mentions

Blog posts, news stories, Wikipedia articles, comments, reviews

Social Media

Tweets, Google+, Facebook likes, shares, ratings

Citations

Web of Science, Scopus, CrossRef, PubMed Central, Microsoft Academic Search

Ref : Altmetrics Manifesto - http://altmetrics.org/about/

Altmetrics

Altmetrics data is aggregated from many sources

https://springerlink.altmetric.com/details/10218936

https://www.nature.com/nclimate/journal/vaop/ncurrent/nclimate3316/metrics

Researchers are communicators:

Within academia:

Presentations and seminars

Academic books

Journal articles and posters

Term papers and essays

Meetings and conferences

Within society:

Speaking at public events

Interviews and news mentions

Press, social media, blogs

We need to measure both?

Role of Libraries and Librarians

High quality content in the form of E-Resources; includes the citation databases and Research Supporttools, Data management tools, Reference management tools (Mendeley, Endnote, zotaro, etc),Grammarly, etc

Help in selecting a good impact factor journal to publish their research

Help in providing list of quality journals in respective disciplines to researchers

Help in Publishing in open access journals/platforms

Help in resolving APC related issues

Help in creating the ORCID profile, etc

Increase the visibility and enhance the impact of their research work - like IRINS

Academic/Creative Writing & Publishing

Awareness on Predatory publishing/ plagiarism

Bibliometrics support

Research Guides and E-Learning Support

Library Outreach activities

Acknowledgement and declaration by presenter

I would like express my sincere thanks to various internet

resources and also Authors of those Internet sources used to

prepare this presentation.

Wherever possible the links have been provided however any

omission is duly regretted.

The presentation is mainly prepared to create an awareness

amongst students and researchers about the various Research

Support Tools.

These slides have been/being used in my various talks and

presentations both online and offline