ALTMETRICS:Rethinking and Exploring New Ways of Measuring Research Outputs

Theng Yin Leng, PhDPI and Director, Centre for HEalthy & Sustainable CitieS (CHESS)Wee Kim Wee School of Communication & InformationNanyang Technological UniversitySingapore

7 October 2015

Acknowledgements This research is supported by

the National Research Foundation, Prime Minister’s Office, Singapore under its Science of Research, Innovation and Enterprise programme (SRIE Award No. NRF2014-NRF-SRIE001-019).

Project Duration: Feb 2015 to Jan 2018

Singapore

Downtown Singapore

Public Housing

Nanyang Technological University

ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs

Challenges in Measuring Research Impact Outputs

“The widely held notion that high-impact publications determine who gets academic jobs, grants and tenure is wrong.” (Michael Eisen 2012 http://goo.gl/t23fpr )

ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs

What is the BIG question?

It seems quite obvious that the impact factor is not the best or even a good measure for the quality of a scientific article, a researcher or a research program. But it will take a concerted effort by universities, funding bodies, and researchers themselves to change habits. A collective change in evaluation procedures and personal behavior in academic publishing could boost the development towards the right direction. And it will take new experiments and new tools to develop optimal quality metrics. Both researchers and librarians have a powerful key role in this movement and I am interested how seriously and quickly they will make use of this role in the near future.

“… it will take a concerted effort by universities, funding bodies, and researchers themselves to change habits.”

http://blog.scienceopen.com/2013/12/journal-impact-factors-time-to-say-goodbye/

ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs

"In the future, tools like these may allow researchers to keep a “live CV” showing up-to-date indicators of their works’ impact next to each product they have produced. Funding organizations could keep timely portfolios of their grants impact across different populations and communities. Tenure, promotion, and hiring committees could assemble research teams whose members have different sets of “impact specialties,” just as sport managers might assemble teams of players with different physical and mental skill sets.

Altmetrics also have great potential for supporting personalized recommender systems; citation-based versions of these have been created before [92], but suffer from the long lag between an idea’s insemination and citation; altmetrics could solve this problem. Sufficiently advanced recommendation tools allow the technical act of publishing to be decoupled from relevance and quality filtering [93], allowing every researchers to essentially consume their own private journal, peer reviewed organically by their scholarly network."

arXiv:1203.4745v1 [cs.DL] 20 Mar 2012Altmetrics in the Wild: Using Social Media to Explore Scholarly Impact Jason Priem*, Heather A. Piwowar**, Bradley M. Hemminger*

5

National Information Standards Organisation (NISO) White Paper (June 2014)

Literature Survey: Papers (2010 – 2014)

TRADITIONAL METRICS

We inputted the titles, abstracts and index retrieved from Google Scholar and Scopus.

Key words used: bibliometrics, analysis, citation, research, medical, journal, article, impact, publication, factor, information, literature, science, topic, publishing, periodicals, humans, index, scientific, human

ALTMETRICS

We inputted the titles, abstracts and index retrieved from Google Scholar and Scopus.

Key words used: altmetrics, research, impact, social, citation, scientific, metrics, analysis, scholarly, data, media, publications, web, article, indicators, mendeley, articles, study, academic

Distribution of Papers in Altmetrics

1. Source: Das, A. K., & Mishra, S. (2014). Genesis of Altmetrics or Article-level Metrics for Measuring Efficacy of Scholarly Communications: Current Perspectives.Available at SSRN 2499467.

[pubs]

ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs

Project Overview

ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs

Specific objectivesa. To carry out a literature survey of traditional metrics and new metrics, that is, altmetrics to measure

research outputs.b. To develop a framework and a set of hypotheses for cross-metric validation.c. To develop new metrics on research impact that can be harvested from social media platforms (e.g.

Twitter, Weibo, etc.).d. To develop new research impact metrics for the different disciplines, broadly divided into: (i) “hard

sciences” research which include sciences, computing and engineering; and (ii) “non-hard sciences” research humanities, arts, business and social sciences.

e. To develop new metrics to measure research innovation and commercialisation.

Deliverablesa. List of altmetrics to measure research outputs derived from social media, and applied to “hard

sciences” disciplines versus “non-hard sciences” research. b. Cross-metric validation findings to identify the effectiveness of the altmetrics compared to

traditional metrics. c. ARIA, an altmetrics system prototype, for researchers, universities, research institutes and policy-

makers. d. User interface guidelines for generating alternative interfaces to the ARIA prototype, supported by

extensive evaluation.e. Publications in internationally reputable journals and conferences.f. Training of research manpower including graduate students and other research staff.g. Organisation of a workshop on “Altemetrics” in Year 3.

ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs

Information Studies

Prof. THENG Yin Leng (PI), WKWSCI, NTU

Ast/Prof. Joanna SIN (Co-PI), WKWSCI, NTU

Ms Samantha ANG, Collaborator

Mr CHIA Yew Boon, Collaborator

NTU Library

Communication Studies

Prof. Rich Ling (Co-PI), WKWSCI, NTU

Ast/Prof. Winson PENG (Co-PI), WKWSCI, NTU

Computer Engineering

Ast/Prof. WEN Yonggang (Co-PI), SCE, NTU

A/Prof. SUN Aixin (Co-PI), SCE, NTU

Ast/Prof. XIAO Xiaokui (Co-PI), SCE, NTU

Prof. Michael KHOR (Co-PI), PO, NTU

NTU ResearchOffice

Prof. Edie RASMUSSEN (Co-PI), UBC, Canada

External Investigator/Advisor

Mendeley Elsevier

Industry Partners

Industry Collaboration & Relevance

NTU is a development partner with Elsevier since Jun 2014 and for the next 2 years.

Hence, the Research Team will be working closely with NTU Research Office and NTU Library in collaboration with Elsevier and Mendeley to test and implement tools and metrics developed.

ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs

ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs

Figure 1a. Interactions for WPs 1-5 leading to WP6

ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs

System Architecture of ARIA prototype (in WP6)

Pilot Work & Demo [click]

NOVELTY Scenarios Of Use (A Multi-Disciplinary Approach)Cross-Metric Validation (Traditional Metrics vs Altmetrics)Framework and ARIA prototype

17

Scenario 1: Journal Level Scenario 2: Article Level Scenario 3: Individual Level

[more]

SUMMARY

• Motivation/Problem Statement: Research outputs have become very diverse and are increasingly being communicated and disseminated via Social Media. Traditional impact indicators, such as citation counts, are slow in measuring impact, they do not cover new and diverse research outputs, nor do they account for a wider non-academic audience.

• Aim/Objectives: Investigate new approaches offered by Social Media to rethink and explore new ways to measure research outputs by comparing traditional metrics with Altmetrics (new metrics based on social media).

• Deliverables: Design, develop and evaluate a framework and system prototype: “Altmetrics for Research Impact Actuation” (ARIA), offering new ways of computing altmetrics to measure research impact.

• Novelty/Contribution to Research and Practice: Cross-metric validation across different disciplines including “hard sciences”, “non-hard sciences”, research innovation and commercialization.

• Beneficiary: Researchers, universities, research institutions, policy-makers, industry and market tie-ups.

• Scenario of use: Exploring altmetrics and comparing to traditional metrics.

Altmetrics Providers:

Almetrics Data Sources:

The Blind Men and the Elephant

Multi-disciplinary and Inter-disciplinary Efforts Required

Altmetrics research: It’s also about good and responsible science & communication.

About ….. Stories of faculty & researchers and their impact (for P&T, CVs, translational research for societal and economic impact, etc)

Contact Details Prof. Theng Yin Leng

Principal InvestigatorDirector, Centre for HEalthy & Sustainable CitieS (CHESS)Wee Kim Wee School of Communication & Information Nanyang Technological University, SingaporeEmail: tyltheng@ntu.edu.sgTel: (65) 6790-5834

Dr. Helen Mojisola ErdtResearch FellowCentre for HEalthy & Sustainable CitieS (CHESS)Wee Kim Wee School of Communication & Information Nanyang Technological University, SingaporeEmail: Mojisola.Erdt@ntu.edu.sg

Thank You

More Info: CHESS

http://www.wkwsci.ntu.edu.sg/Research/Pages/Research-Centres.aspx

http://www.chess.wkwsci.ntu.edu.sg

Altmetrics Project http://

www.wkwsci.ntu.edu.sg/Research/Pages/Altmetrics-Rethinking-and-Exploring-New-Ways-of-Measuring-Research.aspx

http://www.altmetrics.ntuchess.com

Additional Slides

ALTMETRICSRethinking and Exploring New Ways of Measuring Research Outputs

Theng (lead), Wen, Rasmussen, Sin, Ling, RSO, Library

Xiao (lead), Sun, Peng, Ling, Theng

Sun (lead), Peng, Xiao, Khor, Theng

Peng (lead), Ling, Sin, Rasmussen, Theng

Wen (lead), Theng, Khor, Ling, Peng, Rasmussen

Theng (lead), Wen, Sun, Xiao, Sin, Rasmussen, RSO, Library

WP1: Motivation

• With the proliferation of scientific databases, new metrics (or altmetrics) are emerging to indicate research impact. Traditionally, citation is one key impact metric. Lately, a new metric, that is, the number of downloads (Bollen, Van de Sompel, Smith, & Luce, 2005), has started to attract attention. While there has been some research or analysis on factors that influence paper citations, there seems to be a lack of understanding of the relationships between, for example, citations and downloads.

• In WP1, we aim to survey and compile a comprehensive list of traditional metrics (e.g. citation counts, impact factors, etc.) and altmetrics (e.g., opinions, social impact, etc.). We will then formulate a framework to conduct cross-metric validation, and ideally identify factors that could help to enhance research impact.

[back]

WP1: Approach

• Objectives:– To survey and compile a comprehensive list of traditional metrics (e.g. citation counts,

impact factors, etc.) and altmetrics (e.g., opinions, social impact, etc.).– To formulate a framework to conduct cross-metric validation, and ideally identify factors

that could help to enhance research impact.

• Deliverables: – One conference paper and one journal paper.– A list of traditional metrics and altmetrics to measure research outputs derived from

social media, and applied to “hard sciences” disciplines versus “non-hard sciences” research.

– A framework to conduct cross-metric validation, and ideally identify factors that could help to enhance research impact.

[back]

WP2: Motivation

• Existing approaches (for impact actuation based on social media) rely on ad hoc statistics, e.g.,– the number of times that an article is mentioned on Twitter (the

higher the better)– the number of times that an article is downloaded

• However, such statistics are rather inaccurate, e.g.,– a bad article could be mentioned a large number of times on Twitter,

but in a negative way– when people mention an article, they may be simply re-tweeting

others’ posts, without understandings of what the article is actually about

– one can easily increase the number of times that an article is mentioned, by tweeting it repeatedly

• Therefore, we need a more systematic approach to evaluate research impact based on social media

[back]

WP2: Approach• Collect large-scale data about people’s discussions of scientific

articles on social media• Based on the data collected, develop mathematical models

(referred to as information propagation models) about how research-related information is propagated in the social media– A toy example of models:

• If an article has high impact, then it has X% probability to be mentioned by at least one high-profile tweet user, after which it has Y% probability to be re-tweeted by Z number of other tweet users, and the re-tweets have P% probability to have positive sentiments

• Based on our models, devise methods to quantify the impact of articles, based on data from social media– A toy example:

• We observe that an article is mentioned by 2 high-profile tweet users in the first month after its publication, after which it has re-tweeted by 100 other tweet users, but half of the re-tweets are negative. Based on our model, how likely is this article a high-impact work?

[back]

WP3: Motivation• WP3 will focus on metrics to measure impact actuation for “hard

sciences” research which includes sciences, computing and engineering. For example, one would like to understand the Societal Impact of Emerging Topics and Leading Researchers by looking into the life cycle of a specific technology or research. Another example would be the Research on Investment Estimation for National Research Programmes. One interesting metric, the entrepreneurial impact, was exemplified by a study that found that the total market capital of all the companies run or funded by living MIT alumni in 2009 is equivalent to the eleventh economy in the world (Roberts and Eesley, 2009).

• In WP3, we would like to explore further different domains to establish alternative metrics for a local university like the Nanyang Technological University (NTU) as a test-bed, working closely with the NTU Research Support Office (RSO) and NTU Libraries.

[back]

WP3: Approach

• Objectives:– To discover and propose a list of altmetrics for the “hard sciences” disciplines.– Working with NTU RSO and Libraries, we will conduct cross-metric validation tests to

investigate the effectiveness of the altmetrics compared to traditional metrics for the “hard sciences” disciplines.

• Deliverables: – One conference paper and one journal paper.– A software tool to compute the altmetrics for the “hard sciences” disciplines.

[back]

WP4: Motivation• Digital technologies promise new forms of space for action and

interaction by individuals and communities, and open up new avenues of research making it possible to answer questions that were previously unanswerable with conventional and traditional research metrics on “non-hard sciences” research which include humanities, arts, business and social sciences. For example, one would like to understand the impact of a piece of work on society.

• WP4 will focus on quantifying research impact on the “soft sciences” disciplines such as measuring the impact of investments in the arts.

[back]

WP4: Approach

• Objectives:– To discover and propose a list of altmetrics for the “soft sciences” disciplines.– Working with NTU RSO and Libraries, we will conduct cross-metric validation tests to

investigate the effectiveness of the altmetrics compared to traditional metrics for the “soft sciences” disciplines.

• Deliverables: – One conference paper and one journal paper.– A software tool to compute the altmetrics for the “soft sciences” disciplines.

[back]

WP5: Motivation• In technology development, patent statistics are important innovator

indicator to predict impact of technology that would transform the industry, leading to commercialisation. With the access to a huge volume of data, similar techniques could be applied to predict research trends in scientific pursuit.

• In WP5, advanced machine learning techniques can be used to crunch the available data and better understand the model of research innovation. It can be further explored to predict future research trends, and important to the universities, research institutes and policy-makers.

[back]

WP5: Approach• Objectives:

– To discover and propose a list of altmetrics for innovation and commercialisation.– Working with NTU RSO and Libraries, we will conduct cross-metric validation tests to

investigate the effectiveness of the altmetrics compared to traditional metrics for the innovation and commercialisation.

– Using advanced machine learning techniques, we will develop a model of research innovation and a software tool to predict and visusalise future research trends.

• Deliverables: – One conference paper and one journal paper.– A software tool to compute the altmetrics for innovation and commercialisation.– A software tool to model, predict and visualise research innovation and

comercialisation.

[back]

WP6: Motivation• Based on findings from WPs 1-5 and the software tools developed, WP6

will focus on the design, development and evaluation of a system to allow users to obtain altmetrics on research impact actuation.

• Using established usability inspection techniques, for example, heuristic evaluation and cognitive walkthrough, and use cases (or scenarios of use), we will conduct evaluation on the ARIA prototype in order to “gather information about the usability or potential usability of a system to improve features within an interface and its supporting material or to assess a completed interface” (Dix et al., 1997). This involves observing and interviewing 15-20 subjects.

• Findings from our proposed suite of WPs in WP5 will form the basic understanding of user issues pertinent to the design of usable and useful elements on the displays (interfaces) of a self-management system to obtain measurements of research outputs.

[back]

WP6: Approach• Objectives:

– To design and develop an altmetrics system prototype called ARIA for researchers, universities, research institutes and policy-makers.

– Using extensive user studies, we will evaluate alternative interfaces to the ARIA prototype for different users and stakeholders.

• Deliverables: – One conference paper and one journal paper.– ARIA prototype with the respective software tools developed in WPs 2-5 to compute

altmetrics to complement traditional metrics to measure research impact.

[back]