altmetrics - measuring the impact of scientific activities

Background image: http://commons.wikimedia.org/wiki/File:Water_drop_impact_on_a_water-surface_-_(5).jpg

Altmetrics Measuring the impact of scientific activities

Kim HolmbergResearch Unit for the Sociology of Education

University of Turku

(e) kim.j.holmberg@utu.fi

(w3) http://kimholmberg.fi

@kholmber

Oulu, 8.5.2015

Bibliometricsthe study of quantitative aspects of the production, dissemination, and use of recorded information

Tague-Sufcliffe, 1992

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

201420132012201120102009200820072006200520042003200220012000199919981997199619951994

The increase of publications about bibliometrics as indexed by Web of Science between 1994 and 2014 (data retrieved from Web of Science on 9 February, 2015).

Goodhart’s Law: When a measure becomes the target, it ceases to be a good measure

Goodhart, 1975

Outreach and Impact Leader, - Ref:1455401

“ We are seeking to recruit … to support, develop and promote outreach and impact activities …Our aim is to maximize the widest possible impact of the research programme in whichever way is most appropriate … The salary range will be £41,430 - £48,873 per annum...”

Taking impact seriously

Problems with citation-based indicators

Problem 1: Time

0 2 4 6 8 10 12 14 16 18 20

Impact factor window

Immediacy index window

Cited half-life

50% ofcitations

50% ofcitations

Time after publication (Years)

Cit

atio

ns

0 2 4 6 8 10 12 14 16 18 20

Time after publication (Years)

Cit

atio

ns

Time of research assessment

Option 1: Journal impact factor, Julkaisufoorumi, …

Potentialfutureimpact

0 2 4 6 8 10 12 14 16 18 20

Time after publication (Years)

Cit

atio

ns

Time of research assessment

Option 2: Citations (after n years)

Pastsuccess and

impact

0 2 4 6 8 10 12 14 16 18 20

Time after publication (Years)

Cit

atio

ns

How can we measureimpact in realtime?

Problem 2: Journals

1

vs.

Fenner, M. & Lin, J. (2013). Altmetrics in Evolution: Defining & Redefining the Ontology of Article-Level Metrics. http://dx.doi.org/10.3789/isqv25no2.2013.04

2

Problem 3: Impact

Science

IMPACT ON:

Science

Society

Environment

Culture

Economy

Education

IMPACT ON:

SOLUTION?

Altmetricsthe study and use of scholarly impact measures based on activity in online tools and environments

Priem, J. (2014). Altmetrics. In Cronin, B. & Sugimoto, C.R. (Eds.). Beyond Bibliometrics: Harnessing MultidimensionalIndicators of Scholarly Impact. MIT Press, 2014.

Created byresearchers

Created bythe public

Altmetrics

Created byresearchers

Indicatingfuture

scientificimpact?

Altmetrics

Indicatingother types of impact, such

as societalimpact?

Created bythe public

Altmetrics

Researchers

The public

Rank Title Altmetric score News Blogs Tweets

1 Experimental evidence of massive-scale emotional

contagion through social networks

5044 301 130 3801

2 Variation in Melanism and Female Preference in Proximate

but Ecologically Distinct Environments

4823 4 11 6669

3 Artificial sweeteners induce glucose intolerance by

altering the gut microbiota

3500 127 34 3303

4 Stimulus-triggered fate conversion of somatic cells into

pluripotency

2986 159 92 3295

5 Dogs are sensitive to small variations of the Earth's

magnetic field

2956 54 33 2668

6 Christmas 2013: Research: The survival time of chocolates

on hospital wards: covert observational study

2734 3 4 3084

7 Epidemiological modeling of online social network

dynamics

2392 156 19 1649

8 Searching the Internet for evidence of time travelers 2246 34 22 2262

9 Conscious Brain-to-Brain Communication in Humans Using

Non-Invasive Technologies

2160 82 32 1660

10 Were James Bond's drinks shaken because of alcohol

induced tremor?

2146 127 14 1559

Top 10 articles according to the Altmetric Score (Source: http://www.altmetric.com/top100/2014/)

0

50

100

150

200

250

300

350

0

1000

2000

3000

4000

5000

6000

7000

8000

1 2 3 4 5 6 7 8 9 10

Altmetric score

Tweets

News

Blogs

The volume of tweets, blog posts, and news stories for the top ten research articles from 2014 as measured with altmetrics, and their Altmetric Score (number of news stories and blog posts are depicted on the secondary axis to the right in the figure)

Researchers

The public

Low

imp

act

Hig

him

pac

t

Altmetrics

Provide a more nuancedimage of the total impactthat research has made?

Citations Altmetrics

Altmetrics are not alternatives to citations

Citations Tweets

Older publications Newer publications

The effects of time on the volume of citations and tweets (Haustein, Peters, Sugimoto, Thelwall, & Larivière, 2014)

The effects of time

Heterogeneity of altmetrics

doiDigital Object Identifier

Most altmetrics identified from:

Bollen J, Van de Sompel H, Hagberg A, Bettencourt L, Chute R, Rodriguez MA, et al. (2009) ClickstreamData Yields High-Resolution Maps of Science. PLoS ONE 4(3): e4803. doi:10.1371/journal.pone.0004803

Disciplinary differences

Bots?Haustein, S., Bowman, T.D., Holmberg, K., Tsou, A., Sugimoto, C.R. & Lariviére, V. (in press). Tweets as impact indicators: Examining the implications of

automated “bot” accounts on Twitter. To appear in JASIST. arXiv version available at: http://arxiv.org/ftp/arxiv/papers/1410/1410.4139.pdf.

Gaming altmetrics?

Stories behind the altmetrics?

Altmetrics and open science?

Measuring the Societal Impact of Open ScienceAvoimen Tieteen Yhteiskunnallisen Vaikuttavuuden Mittarit

Project is financed by the Ministry of Education and Culture in Finland (2015-2016)

Goals of the research project:1) Investigate altmetrics (at country-level, i.e.

Finland) and create a prototype to demonstrate them

2) Develop methods to map societal impact of research (and to analyze the stories behindthe altmetrics)

Shameless selfpromotion:

Bookforthcominglater in 2015

Kim Holmbergkim.j.holmberg@utu.fihttp://kimholmberg.fi@kholmber

Kiitos!

Slides available athttp://www.slideshare.net/kholmber