altmetrics journal club

of 35 /35
Thelwall, Mike, et al. " Do altmetrics work? Twitter and ten other social web services. " PloS one 8.5 (2013): e64841. Anne Madden Journal Club 27 th April 2015

Author: anne-madden

Post on 16-Jul-2015




0 download

Embed Size (px)


Page 1: Altmetrics journal club

Thelwall, Mike, et al. "Do altmetrics work? Twitter and ten other social web services." PloS

one 8.5 (2013): e64841.

Anne Madden

Journal Club 27th April 2015

Page 2: Altmetrics journal club

• What are altmetrics?

• How are they calculated?

• Paper: Rationale and methodology

• Results?

• Discussion: do altmetrics “work”?

Page 3: Altmetrics journal club

Priem (2010): definition

Page 4: Altmetrics journal club

“No one can read everything. We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped.”From the altmetrics manifesto. (Priem 2010)

Page 5: Altmetrics journal club

Page 6: Altmetrics journal club

Rationale for this study

“To what extent do the altmetric indicators associate with citation


Building some form of evidence base to underpin altmetrics?

Page 7: Altmetrics journal club
Page 8: Altmetrics journal club

Methodology• PubMed articles with a non-zero altmetric


• Sign test: “..each article is compared only against the two articles published immediately before and after it….”

• Self-citations removed – why?

• Altmetric scores gathered between July 2011 – 1st Jan 2013

Page 9: Altmetrics journal club

Definition of success

“..if the altmetric score is higher than the average altmetric score of the two adjacent articles and its citation score is higher than the average of the two adjacent articles OR the altmetric score is lower than the average altmetric score of the two adjacent articles and its citation score is lower than the average of the two adjacent articles.”

Page 10: Altmetrics journal club


1. Total no. of successes v no. of failures: overall success2. No. of journals where success exceeds failure: success3. Individual altmetric level: mixed results. Four

altmetrics “significantly and positively correlate with citations..the correlation for Twitter is significant and negative”.

• “..results provide strong evidence that six of the eleven altmetrics.. associate with citation counts. ..correlation effect size is unknown”

Page 11: Altmetrics journal club

Experiment. Starting from non-zero citations: a very rough comparison

• PubMed papers, with citations

• Timeframe: mid to end 2013

• Search: “Influenza” [Title]

• 20 papers selected at random

• 14 had non-zero citations

Page 12: Altmetrics journal club
Page 13: Altmetrics journal club
Page 14: Altmetrics journal club

Lowest altmetric score was for non-English paper (Chinese)

Cut-off point where association between altmetric score and citation count lose significance.

Page 15: Altmetrics journal club

Boon 2014, IATUL Proceedings: Altmetrics is an Indication of Quality Research or Just HOT Topics

Page 16: Altmetrics journal club

Discussion• Anonymity in most altmetric tools• Multiple accounts: the “Scholarly

Selfie”• “Gaming” – could affect both

altmetric and citation scores• Relative labour-intensity and

engagement with paper

Page 17: Altmetrics journal club
Page 18: Altmetrics journal club


Page 19: Altmetrics journal club
Page 20: Altmetrics journal club

Groups doing it for themselves

“Early experience with JACR tweet chats demonstrates that organizing Twitter microblogging activities around topics of general interest to their target readership bears the potential for medical journals to increase their audiences and reach.”

Hawkins, C. Matthew, et al. "The Impact of Social Media on Readership of a Peer-Reviewed Medical Journal." Journal of the American College of Radiology 11.11 (2014): 1038-1043.

Page 21: Altmetrics journal club

Topical, timely, authoritative, clear message, relevance to a broad audience

Appeals to a broader audience; the ready-made headline…

4 Wikipedia pages (aubergine bit)

Page 22: Altmetrics journal club

Do altmetrics work?

“Not everything that counts can be counted, and not everything that can be

counted counts”

Page 23: Altmetrics journal club

Neither altmetrics nor citation scores are

evidence of quality.

Page 24: Altmetrics journal club
Page 25: Altmetrics journal club

“Although association preferences documented in our study theoretically could be a consequence of either mating or shoaling preferences in the different female groups investigated (should we cite the crappy Gabor paper here?), shoaling preferences are unlikely drivers of the documented patterns ……”

Page 26: Altmetrics journal club
Page 27: Altmetrics journal club

From Ben Goldacre on YouTube

Page 28: Altmetrics journal club

Hopewell, Sally, et al. "Publication bias in clinical trials due to statistical significance or direction of trial results." The

Cochrane Library (2009).

“Trials with positive findings are more likely to be

published and published quicker than trials with

negative findings.”

Failure to publish: “scientific misconduct” valid

expectation on the part of trial participants that

their participation will contribute to knowledge.

Page 29: Altmetrics journal club

…which scholarly products are read, discussed, saved and recommended as well as cited.

“..provide an insight into things we have not measured before….” Tattersall 2015

“…analyzing the impact of outputs in different formats.. as opposed to the analysis of only journal papers.” (Costas 2014)

“…impact of scholarly material, with an emphasis on social media outlets…” (Priem 2014)

…more timely data, showing evidence of impact in days instead of years

..the impact of web-native scholarly products

…impacts on diverse audiences including scholars but also practitioners, clinicians, educators and the general public. (Piwowar 2013)

Page 30: Altmetrics journal club

Scholarly journals or scholarly communication?

“For all new grant applications from 14 January, the US National Science Foundation (NSF) asks a principal investigator to list his or her research “products” rather than “publications” in the biographical sketch section. This means that according to the NSF, a scientist’s worth is not dependent solely on publications.”

Piwowar, Heather. "Altmetrics: Value all research products." Nature 493.7431 (2013): 159-159.

Page 31: Altmetrics journal club

Djuricich, Alexander M., and Janine E. Zee-Cheng. "Live tweeting in medicine:'Tweeting the meeting'." International Review of Psychiatry 0

(2015): 1-7.

Page 32: Altmetrics journal club

Cochran, Amalia, et al. "Use of Twitter to document the 2013 Academic Surgical Congress." Journal of Surgical

Research 190.1 (2014): 36-40.

Page 33: Altmetrics journal club

Aslam, Anoshé A., et al. "The Reliability of Tweets as a Supplementary Method of Seasonal Influenza Surveillance." Journal of medical Internet research 16.11 (2014).

“….this study demonstrated increased accuracy in using Twitter as a supplementary surveillance tool for influenza as better filtering and classification methods yielded higher correlations for the 2013-2014 influenza season than those found for tweets in the previous influenza season, where emergency department ILI rates were better correlated to tweets than sentinel-provided ILI rates.”

Page 34: Altmetrics journal club

• H-index outdated? Register for your ORCID Identifier ( For more info see:

• Create an author profile on Google Scholar Citations (

• Create an impactstory account for your biog. (

From: Rathemacher, Andrée J. "Article-Level Metrics and Altmetrics: New Ways to Measure the Impact of Your Research." (2014).

• When publishing ask for permission to store a copy in the Institutional Repository (

Page 35: Altmetrics journal club

“Do altmetrics work?” Has Thelwall proven his case?

• Altmetrics were never intended to replace / anticipate / mimic citation counts.

• Altmetrics have a far wider scope, with more focus on unpublished born-digital material and other audiences. Therefore, to prove they “work”, more multi-faceted research required.