scholarly communication and evaluation: from bibliometrics to altmetrics stefanie haustein...

34
Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein [email protected] @stefhaustein crc.ebsi.umontreal.ca/sloan

Upload: byron-porter

Post on 19-Dec-2015

221 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Scholarly communicationand evaluation:from bibliometrics to altmetrics

Stefanie [email protected]@stefhausteincrc.ebsi.umontreal.ca/sloan

Page 2: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Scholarly Communication• peer-reviewed journals

1665: Journal de Sçavans

Philosophical Transactions

replace personal correspondences

• registration

• certification

• dissemination

• archiving

• “Little Science, Big Science”Derek J. de Solla Price (1963)

exponential growth

Page 3: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Scholarly Communication• citation analysis as retrieval tool to handle

information overload“It would not be excessive to demand that the thorough

scholar check all papers that have cited or criticized such

papers, if they could be located quickly. The citation index

makes this check practicable.”

• citation analysis as evaluation method oversimplification of scientific work and success

publications = productivity | citations = impact

adverse effects

Garfield, 1955, p. 108

Page 4: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Scholarly Communication• digital revolution

electronic publishing

• acceleration, openness and diversification of scholarly output and impact open access and open science

• altmetrics manifesto:

Priem, Taraborelli, Groth and Neylon (2010)

“No one can read everything. We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped. However, the growth of new, online scholarly tools allows us to make new filters; these altmetrics reflect the broad, rapid impact of scholarship in this burgeoning ecosystem.”

Page 5: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

AltmetricsCriticism against current form of research evaluation:• peer-reviewed publications in scholarly journals as the only

form of output that “counts”• particularly against Journal Impact Factor

• citations as the only form of impact that “counts”

Altmetrics as alternatives:• including all research “products”

• similar but more timely than citations predicting scientific impact

• different, broader impact than citations measuring societal impact

Page 6: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Altmetrics• alternative use and visibility of publications

on social media:

more traditional forms of use:

• alternative forms of research output

pragmatic development based on IT developments

Page 7: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Definitions and terminology• webometrics

“Polymorphous mentioning is likely to become a defining feature of Web-based scholarly communication.”

“There will soon be a critical mass of web-based digital objects and usage statistics on which to model scholars’ communication behaviors […] and with which to track their scholarly influence and impact, broadly conceived and broadly felt.”

• PLOS article level metrics (ALM)

• altmetrics“study and use of scholarly impact measures based on activity in online tools and environments”

“a good idea but a bad name”

Priem (2014, p. 266)

Cronin, Snyder, Rosenbaum, Martinson & Callahan (1998, p.1320)

Cronin (2005, p. 196)

Rousseau & Ye (2013, p. 2)

Page 8: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Definitions and terminology

informetrics

scientometrics

bibliometrics

cybermetrics

webometrics altmetrics

adapted from: Björneborn & Ingwersen (2004, p. 1217)

Page 9: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Definitions and terminology

adapted from: Björneborn & Ingwersen (2004, p. 1217)

informetrics

scientometrics

bibliometrics

cybermetrics

webometrics social media metrics

social media metrics

Haustein, Larivière, Thelwall, Amyot & Peters (2014)

Page 10: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Definitions and terminology

adapted from: Björneborn & Ingwersen (2004, p. 1217)

informetrics

scientometrics

bibliometrics

cybermetrics

webometrics social media metrics

social media metrics

“Although social media metrics seems a better fit as an umbrella term because it addresses the social media ecosystem from which they are captured, it fails to incorporate the sources that are not obtained from social media platforms (such as mainstream newspaper articles or policy documents) that are collected (for instance) by Altmetric.com.“Haustein, Bowman & Costas (2015, p. 3)

Page 11: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Definitions and terminology

adapted from: Björneborn & Ingwersen (2004, p. 1217)

informetrics

scientometrics

bibliometrics

cybermetrics

webometrics social media metrics

scholarly metrics

Page 12: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Definitions and terminology

adapted from: Björneborn & Ingwersen (2004, p. 1217)

informetrics

scientometrics

bibliometrics

cybermetrics

webometrics social media metrics

scholarly metrics

scholarly metrics

“[T]he heterogeneity and dynamicity of the scholarly communication landscape make a suitable umbrella term elusive. It may be time to stop labeling these terms as parallel and oppositional (i.e., altmetrics vs bibliometrics) and instead think of all of them as available scholarly metrics—with varying validity depending on context and function.“Haustein, Sugimoto & Larivière (2015, p. 3)

Page 13: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Definitions and terminologyActs leading to (online) events used for metrics

RESEARCH OBJECT

Hau

ste

in,

Bo

wm

an &

Co

stas

(20

15)

Page 14: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Social media metrics: research

• Which social media metrics are valid impact indicators?

• What kind of impact do the various metrics reflect?

• What is the relationship between social media activity and bibliometric variables?

• Which content receive the most attention on the platforms?

• Who is engaging with scholarly material on social media sites?

• What are the motivations behind this use?

Page 15: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Prevalence: social media uptake

• social media activity around scholarly articles grows 5% to 10% per month (Adie & Roe, 2013)

• Mendeley and Twitter largest sources for mentions of scholarly documents:

Mendeley 521 million bookmarks2.7 million users32% increase of users from 9/2012 to

09/2013 (Haustein & Larivière,

2014)

Twitter 500 million tweets per day230 million active users39% increase of users from 9/2012 to

09/2013ca. 10% of researchers in professional

context

Page 16: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Prevalence: coverage

Mendeley93% of Science articles 2007 (Li, Thelwall & Giustini, 2012)

94% of Nature articles 2007 (Li, Thelwall & Giustini, 2012)

80% of PLOS journals papers 2003-2010 (Priem, Piwowar & Hemminger, 2012)

66% of PubMed/WoS papers 2010-2012 (Haustein et al., 2014a)

63% of WoS papers with DOIs 2005-2011 (Zahedi, Costas & Wouters, 2014)

47% of Social Science WoS papers 2008 (Mohammadi et al., 2014)

35% of Engineering & Techn. WoS papers 2008 (Mohammadi et al., 2014)

31% of Physics WoS papers 2008 (Mohammadi et al., 2014)

13% of Humanities WoS papers 2008 (Mohammadi & Thelwall, 2014)

Twitter 2% of WoS papers with DOIs 2005-2011 (Zahedi, Costas & Wouters, 2014)

9% of PubMed/WoS 2010-2012 (Haustein et al., 2014b)

13% of WoS papers with DOIs July-December 2011 (Costas, Zahedi & Wouters, 2014)

22% of WoS papers with DOIs 2012 (Haustein, Costas & Larivière, 2015)

Page 17: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Prevalence: density

Mean number of events per paper per document typeWoS papers 2012 with DOI

(Haustein, Costas & Larivière, 2015

Page 18: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Prevalence: density / intensity

Mean number of events per paper

WoS papers with DOIs 2012

all papers / papers with at least one social media event

0.03 / 1.51 Blogs

0.78 / 3.65 Twitter

0.08 / 1.78 Facebook

0.01 / 1.66 Google+

0.01 / 1.54 Mainstream media

PubMed/WoS papers 2010-2012

6.43 / 9.71 Mendeley(Haustein et al., 2014a)

(Haustein, Costas & Larivière, 2015)

Page 19: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Similarity: correlations

Spearman correlations with citations

WoS papers with DOIs 2012

all papers / papers with at least one social media event

0.124 / 0.191 Blogs

0.194 / 0.148 Twitter

0.097 / 0.167 Facebook

0.065 / 0.209 Google+

0.083 / 0.199 Mainstream media

PubMed/WoS papers 2011

0.386 / 0.456 Mendeley(Haustein et al., 2014a)

(Haustein, Costas & Larivière, 2015)

Page 20: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Popularity: highly tweeted

Highly tweeted Physics paper

Page 21: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Popularity: highly tweeted

Highly tweeted paper

Page 22: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Popularity: highly tweeted

Highly tweeted paper

Page 23: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Communities of attentionDistinguishing between types of Twitter impact• engagement = dissimilarity with paper title

• exposure = number of followers

Page 24: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Communities of attention• 660,149 original tweets (Altmetric.com up to June 2014)

• 237,222 tweeted documents (WoS 2012 with DOI)

• 125,083 unique users• number of tweets to 2012 papers

• mean tweets per day (all tweets up to April 2015)

• mean relative citation rate of tweeted papers

• mean engagement (dissimilarity between tweet and paper title)

• mean exposure (mean number of followers during tweet)

• mean number of followers (April 2015)

• mean number of following (April 2015)

• tweeted document coupling user network(Haustein, Bowman & Costas, submitted)

Page 25: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Communities of attention

exposure

enga

gem

ent

median dissimilarity with paper title

med

ian

num

ber

of f

ollo

wer

s

influencers / brokers

orators / discussing

disseminators / mumblers

broadcasters

tweet text differs from paper title

tweet text is identical to paper title

few followers many followers

Page 26: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Communities of attention

number of users N = 125,083mean tweets to papers tp = 5.3mean tweets per daytpd = 5.9mean relative citation rate mncs = 2.3mean engagement men = 53.3mean exposure mex = 1,382.6mean number of followers mnfers= 2,027.2mean number of following mnfing= 855.6

exposure

enga

gem

ent

N = 29,770tp = 3.2tpd = 10.1mncs = 2.4men = 74.2mex = 2,876.9mnfers= 4,177.3mnfing = 1,327.1

N = 32,768tp = 1.7tpd = 1.8mncs = 2.5men = 75.8mex = 82.7mnfers = 191.4mnfing = 259.0

N = 32,680tp = 11.5tpd = 9.4mncs = 2.1men = 32.7mex= 2,511.2mnfers = 3,396.8mnfing = 1,497.3

N = 29,865tp = 4.4tpd = 1.7mncs = 2.2men = 30.3mex = 84.6mnfers = 178.0mnfing = 267.4

(Haustein, Bowman & Costas, submitted)

Page 27: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Communities of attention

number of users N = 125,083mean tweets to papers tp = 5.3mean tweets per daytpd = 5.9mean relative citation rate mncs = 2.3mean engagement men = 53.3mean exposure mex = 1,382.6mean number of followers mnfers= 2,027.2mean number of following mnfing= 855.6

exposure

enga

gem

ent

N = 29,770tp = 3.2tpd = 10.1mncs = 2.4men = 74.2mex = 2,876.9mnfers= 4,177.3mnfing = 1,327.1

N = 32,768tp = 1.7tpd = 1.8mncs = 2.5men = 75.8mex = 82.7mnfers = 191.4mnfing = 259.0

N = 32,680tp = 11.5tpd = 9.4mncs = 2.1men = 32.7mex= 2,511.2mnfers = 3,396.8mnfing = 1,497.3

N = 29,865tp = 4.4tpd = 1.7mncs = 2.2men = 30.3mex = 84.6mnfers = 178.0mnfing = 267.4

(Haustein, Bowman & Costas, submitted)

Page 28: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Communities of attention

number of users N = 125,083mean tweets to papers tp = 5.3mean tweets per daytpd = 5.9mean relative citation rate mncs = 2.3mean engagement men = 53.3mean exposure mex = 1,382.6mean number of followers mnfers= 2,027.2mean number of following mnfing= 855.6

exposure

enga

gem

ent

N = 29,770tp = 3.2tpd = 10.1mncs = 2.4men = 74.2mex = 2,876.9mnfers= 4,177.3mnfing = 1,327.1

N = 32,768tp = 1.7tpd = 1.8mncs = 2.5men = 75.8mex = 82.7mnfers = 191.4mnfing = 259.0

N = 32,680tp = 11.5tpd = 9.4mncs = 2.1men = 32.7mex= 2,511.2mnfers = 3,396.8mnfing = 1,497.3

N = 29,865tp = 4.4tpd = 1.7mncs = 2.2men = 30.3mex = 84.6mnfers = 178.0mnfing = 267.4

(Haustein, Bowman & Costas, submitted)

Page 29: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Communities of attention

more than 100 tweeted papers

708 of 125,083 users (0.6%)

9 57

130 512

(Haustein, Bowman & Costas, submitted)

Page 30: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Communities of attention

708 of 125,083 users (0.6%)

more than 100 tweeted papers

(Haustein, Bowman & Costas, submitted)

Page 31: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Some conclusions• citations, Mendeley readers and tweets reflect different

kinds of impact on different social groups• Mendeley seems to mirror use of broader but still

academic audience, largely students and postdocs• Twitter seems to reflect popularity among general

public and represents mix of societal impact, scientific discussion and dissemination and buzz

• differences between disciplines, document types and age reader counts and tweets cannot be directly compared

without normalization

Page 32: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Some conclusions• fundamental differences between social media metrics

and citations:• gatekeeping

• community

• engagement

• quantitative and qualitative research needed:• determine biases and confounding factors

• identify user groups

• identify user motivations and types of use

meaning of social media metrics needs to be understood before they are applied to research evaluation

Page 33: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Some tips

When using altmetrics:

• time biases apply: don’t use for old papers!

• most metrics only captured for DOIs: remember limitation!

• social media metrics do not replace citations: don’t substitute!

• social media metrics are heterogeneous: don’t blend!

• document type: don’t compare!

• disciplinary differences: don’t compare!

• not all events reflect use or impact: differentiate!

• motivations and confounding factors unknown: be careful!

Page 34: Scholarly communication and evaluation: from bibliometrics to altmetrics Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Stefanie Haustein

Thank you for your attention!

[email protected]@stefhausteincrc.ebsi.umontreal.ca/sloan

Thank you for your attention!

Questions?

Thank you for your attention!

Questions?Obrigada!

Special Issue “Social Media Metrics” Aslib Journal of Information Management 67(3)

Early View: www.emeraldinsight.com/toc/ajim/67/3Links to OA preprints: crc.ebsi.umontreal.ca/aslib/