Scientific interactions and research evaluation: from bibliometrics to altmetrics

Stefanie Haustein stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Scientific Interactions

Research Evaluation

Opportunities & Challenges

Invisible Colleges

Père Marin Mersenne (1588-1648)

Henry Oldenburg (1619-1677)

http://commons.wikimedia.org/wiki/File:Marin_mersenne.jpg#/media/File:Marin_mersenne.jpg http://commons.wikimedia.org/wiki/File:Henry_Oldenburg.jpg

Invisible Colleges

Père Marin Mersenne (1588-1648)

Henry Oldenburg (1619-1677)

http://commons.wikimedia.org/wiki/File:Marin_mersenne.jpg#/media/File:Marin_mersenne.jpg http://commons.wikimedia.org/wiki/File:Henry_Oldenburg.jpg

Information Hubs

Scientific Societies

L’Académie royale des sciences 22 December 1666

The Royal Society 28 November 1660

http://upload.wikimedia.org/wikipedia/commons/thumb/6/61/Louis_XIV_und_Colbert_in_der_Akademie.jpg/640px-Louis_XIV_und_Colbert_in_der_Akademie.jpg https://www.rcseng.ac.uk/museums/hunterian/images/lost-museums-2011/the-royal-society-repository/Image%201%20-%20GreshamCollege.jpg/image_preview http://en.wikipedia.org/wiki/Nullius_in_verba#/media/File:Bookplate_of_the_Royal_Society_(Great_Britain).jpg

Scientific Societies

L’Académie royale des sciences 22 December 1666

The Royal Society 28 November 1660

http://upload.wikimedia.org/wikipedia/commons/thumb/6/61/Louis_XIV_und_Colbert_in_der_Akademie.jpg/640px-Louis_XIV_und_Colbert_in_der_Akademie.jpg https://www.rcseng.ac.uk/museums/hunterian/images/lost-museums-2011/the-royal-society-repository/Image%201%20-%20GreshamCollege.jpg/image_preview http://en.wikipedia.org/wiki/Nullius_in_verba#/media/File:Bookplate_of_the_Royal_Society_(Great_Britain).jpg

Institutionalization of Science

Scientific Journals

Le journal des sçavans 5 January 1665

Philosophical Transactions 6 March 1665

http://de.wikipedia.org/wiki/Journal_des_s%C3%A7avans#/media/File:1665_journal_des_scavans_title.jpg http://en.wikipedia.org/wiki/Philosophical_Transactions_of_the_Royal_Society#/media/File:Philosophical_Transactions_Volume_1_frontispiece.jpg

Scientific Journals

Le Journal des sçavans 5 January 1665

Philosophical Transactions 6 March 1665

http://de.wikipedia.org/wiki/Journal_des_s%C3%A7avans#/media/File:1665_journal_des_scavans_title.jpg http://en.wikipedia.org/wiki/Philosophical_Transactions_of_the_Royal_Society#/media/File:Philosophical_Transactions_Volume_1_frontispiece.jpg

Registration Certification

Dissemination Archiving

Scientific Articles: 17th to Early 19th Century •  Experiments and descriptions of the natural world •  Avoiding “fine speaking” •  Various styles of arguing •  Qualitative and personal judgements

Harmon, J.E. & Gross, A.G. (2007). The Scientific Literature. A Guided Tour. Chicago: University of Chicago Press.

Modern Scientific Articles •  Professionalized and highly specialized •  Increased focus on data, graphs, tables and theory •  Impersonal, technical and codified •  Style guides and gatekeeping •  Citations •  Introduction, Methods, Results and Discussion

Larivière, V., Archambault, É. & Gingras, Y. (2008). Long-term variations in the aging of scientific literature: From exponential growth to steady-state science (1900-2004). Journal of the American Society for Information Science and Technology, 59(2), 288-296. Sollaci, L.B. & Pereira, M.G. (2004). The introduction, methods, results, and discussion (IMRAD) structure: a fifty-year survey. Journal of the Medical Library Association, 92(3), 364-371

Sollaci & Pereira (2004, p. 365)

Proportion of IMRaD adoption in medical journals Number of references 1900 to 2004

Larivière, Archambaul & Gingras (2008, p. 293)

arXiv submission statistics from http://arxiv.org/stats/monthly_submissions Larivière, V., Lozano, G.A. & Gingras, Y. (2014). Are elite journals declining? Journal of the Association for Information Science and Technology, 65(4), 649-655.

•  Improved access •  Acceleration

•  Collaboration •  Peer review •  Distribution of preprints

•  Decreasing importance of scientific journals •  Journal functions •  Diversification of publication

venues •  Appearance and symbolic

capital of journals unchanged

Digital Revolution Submissions to arXiv

Share of top 1% most cited papers

Larivière, Lozano & Gingras (2014, p. 652)

Academic Publishing Market

Larivière, V., Haustein, S. & Mongeon, P. (in press). The oligopoly of academic publishers in the digital era. PLoS ONE.

Larivière, Haustein & Mongeon (in press)

•  Aggravation of serials crisis

•  Elsevier: €3,400 subscription price, 30% increase

•  Profit margins of commercial publishers up to 40%

•  Decline of scientific societies as publishers

•  >50% of papers owned by five major publishers

Budapest Open Access Initiative

Open Access

“immediate, free availability on the public internet, permitting any users to read, download, copy, distribute, print, search or link to the full text of these articles”

Budapest Open Access Initiative (2002)

Archambault et al. (2013, p. 13)

Freely available journal papers 2004 to 2011 •  Gold and Green •  Libre and Gratis •  Hybrid journals

•  Elsevier: $500 to 5,000 •  Springer: $3,000 •  Wiley: $3,000

Archambault, É., Amyot, D., Deschamps, P., Nicol, A., Rebout, L. & Roberge, G. (2013). Proportion of Open Access Peer-Reviewed Papers at the European and World Levels 2004-2011. Report for the European Commission. http://www.science-metrix.com/pdf/SM_EC_OA_Availability_2004-2011.pdf

Open Science “opening up the research process by making all of its outcomes, and the way in which these outcomes were achieved, publicly available on the World Wide Web”

Kraker et al. (2011, p. 645)

Kraker, P., Leony, D., Reinhardt, W. & Beham, G. (2011). The case for an open science in technology enhanced learning. International Journal of Technology Enhanced Learning, 3(6), 643-654.

•  Open Data •  Open Source •  Open Methodology •  Open Access •  Open Peer Review

Björneborn, L. & Ingwersen, P. (2004), Toward a basic framework for webometrics. Journal of the American Society for Information Science and Technology, 55(14), 1216–1227.

Definition of Scholarly Metrics

adapted from: Björneborn & Ingwersen (2004, p. 1217)

informetrics

scientometrics

bibliometrics

cybermetrics

webometrics altmetrics

Otlet, P. (1934). Traité de documentation: le livre sur le livre, théorie et pratique. Pritchard, P. (1927). Statistical bibliography or bibliometrics? Journal of Documentation, 25, 348-349..

Definition of Scholarly Metrics

informetrics

scientometrics

bibliometrics

cybermetrics

webometrics altmetrics

“La «Bibliometrie» sera la partie définie de la Bibliologie qui s'occupe de la mesure ou quantité appliquée aux livres.” “the application of mathematics and statistical methods to books and other media of communication”

Pritchard (1969, p. 348)

Otlet (1934, p. 14)

Priem, J. (2014). Altmetrics. In B. Cronin & C. R. Sugimoto (Eds.), Beyond bibliometrics: harnessing multidimensional indicators of performance (pp. 263–287). Cambridge, MA: MIT Press. Rousseau, R. & Ye, F. (2013). A multi-metric approach for research evaluation. Chinese Science Bulletin, 10–12. doi:10.1007/s11434-013-5939-3

informetrics

scientometrics cybermetrics

webometrics altmetrics

“study and use of scholarly impact measures based on activity in online tools and environments” “a good idea but a bad name”

Rousseau & Ye (2013, p. 2)

Priem (2014, p. 266)

bibliometrics

Definition of Scholarly Metrics

Definition of Scholarly Metrics

informetrics

scientometrics

bibliometrics

cybermetrics

webometrics altmetrics

scholarly metrics

Scholarly metrics are metrics based on acts and events (e.g., viewing, reading, saving, diffusing, mentioning, citing, reusing, modifying) related to scholarly documents (e.g., papers, blog posts, datasets, code, notes) or scholarly agents (e.g., researchers, universities, journals).

Gross, P.L.K. & Gross, E.M. (1927). College libraries and chemical education. Science, 66(1713), 385-389..

Citation analysis for objective collection management

Bibliometrics for Library Mangement

Gross & Gross, 1927, p. 386

Journals cited in the Journal of the American Chemical Society 1926

Garfield, E. (1955). Citation indexes for science. A new dimension in documentation through association of ideas. Science, 122, 108-111.

Bibliometrics for Information Retrieval

Retrieval tool to handle information overload

•  Institute for Scientific Information •  Science Citation Index

•  Source Author Index •  Citation Index

“It would not be excessive to demand that the thorough scholar check all papers that have cited or criticized such papers, if they could be located quickly. The citation index makes this check practicable.”

Garfield, 1955, p. 108

Derek J. de Solla Price Science since Babylon (1961) Little Science – Big Science (1963)

Bibliometrics for Sociology of Science

Price, D. J. d. S. (1961). Science Since Babylon. New Haven / London: Yale University Press, Price, D. J. d. S. (1963). Little Science, Big Science. New York: Columbia University Press.

Merton, R. K. (1988). The Matthew effect in science, II: Cumulative advantage and the symbolism of intellectual property. Isis, 79, 606–623.

Robert K. Merton •  Social norms of science

•  Communalism •  Universalism •  Disinterestedness •  Organized skepticism

•  Matthew effect

Bibliometrics for Sociology of Science

“symbolically, [the reference] registers in the enduring archives the intellectual property of the acknowledged source by providing a pellet of peer recognition of the knowledge claim” Merton (1988, p. 621)

•  Performance measurement and policy instrument

•  Commercialization

Bibliometrics for Research Evaluation

“When used properly, bibliometric indicators can provide a ‘monitoring device’ for university research-management and science policy. They enable research policy-makers to ask relevant questions of researchers on their scientific performance, in order to find explanations of the bibliometric results in terms of factors relevant to policy.”

Moed et al. (1985, p. 131)

Moed, H.F., Burger, W.J.M., Frankfort, J.G, van Raan, A.F.J. (1985). The use of bibliometric data for the measurement of university research performance. Research Policy, 14(3), 131-149.

•  Oversimplification publications = productivity citations = impact

•  Uninformed use and misuse Impact Factor h-index

•  Adverse effects “salami” publishing honorary authorship self-citations citation cartels

Bibliometrics for Research Evaluation

Hvistendahl, M. (2013). China’s publication bazaar. Science, 342(6162), 1035-1039. van Noorden, R. (2013). Brazilian citation scheme outed: Thomson Reuters suspends journals from its rankings for ‘citation stacking’, Nature, 500(7464), 510-511.

Criticism against current form of research evaluation Ø  Alternative forms of research ouput

Ø  Alternative use and visibility of publications

Altmetrics

“We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped. However, the growth of new, online scholarly tools allows us to make new filters; these altmetrics reflect the broad, rapid impact of scholarship in this burgeoning ecosystem.”

…

Priem et al. (2010)

Information overload

…

Priem, J., Taraborelli, D., Groth, P., & Neylon, C. (2010). Alt-metrics: a manifesto. October. Retrieved from http://altmetrics.org/manifesto/ Piwowar, H. A. (2013). Altmetrics: Value all research products. Nature, 493(7431), 159.

Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ? It - Information Technology, 56(5), 207–215. (for Mendeley reader counts only: 2010-2012 PubMed/WoS papers) Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830. (for all metrics except Mendeley reader counts: 2012 WoS papers with a DOI)

Altmetrics

What do we know?

%

Altmetrics

What do we know?

Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ? It - Information Technology, 56(5), 207–215. (for Mendeley reader counts only: 2010-2012 PubMed/WoS papers) Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830. (for all metrics except Mendeley reader counts: 2012 WoS papers with a DOI)

%

Altmetrics

What do we know?

Haustein, S., Larivière, V., Thelwall, M., Amyot, D., & Peters, I. (2014). Tweets vs. Mendeley readers: How do these two social media metrics differ? It - Information Technology, 56(5), 207–215. (for Mendeley reader counts only: 2010-2012 PubMed/WoS papers) Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830. (for all metrics except Mendeley reader counts: 2012 WoS papers with a DOI)

%

Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830.

Altmetrics

What do we know? Biomedical & Health Sciences

Social Sciences & Humanities Life & Earth Sciences

Natural Sciences & Engineering Mathematics & Computer Science Tw

itter c

over

age

Docu

ment

type

Haustein, S., Costas, R. & Larivière, V. (2015) Characterizing social media metrics of scholarly papers: The effect of document properties and collaboration patterns. PLoS ONE, 10(5), e0127830.

Altmetrics

What do we know? Biomedical & Health Sciences

Social Sciences & Humanities Life & Earth Sciences

Natural Sciences & Engineering Mathematics & Computer Science Tw

itter c

over

age

Altmetrics

Highly tweeted

Altmetrics

Highly tweeted

Altmetrics

Highly tweeted

Altmetrics

Highly tweeted

Altmetrics

Altmetrics in the wild

Meaning of metrics Opportunities •  Heterogenous users •  Diverse motivations

Challenges •  Understanding underlying processes •  Determining the meaning of metrics

Meaning of metrics Motivations and norms

Saving to Mendeley

Mentioning in News

Meaning of metrics Motivations and norms

Recommending on F1000

Tweeting

Meaning of metrics

Bertin, M., Atanassova, I., Gingras, Y., & Larivière, V. (in press). The invariant distribution of references in scientific articles. Journal of the Association for Information Science and Technology.

Bertin, Atanassova, Gingras & Larivière (in press, p. 13)

Distribution of references along the IMRaD structure

Citing in a journal article

Meaning of Metrics

Haustein, S., Bowman, T.D. & Costas, R. (in press). Interpreting “altmetrics”: viewing acts on social media through the lens of citation and social theories. In Sugimoto , C.R. (Ed.). Theories of Informetrics: A Festschrift in Honor of Blaise Cronin.

Acts leading to (online) events used for metrics

Haustein, Bowman & Costas (in press, p. 13)

RESEARCH OBJECT

Opportunities •  Detailed life cycle of scientific output •  Fine-grained indicators and adequate benchmarks

Challenges •  Versions of research output •  Publication dates

Time Analysis

Time Analysis Journal article

•  Submitted manuscript •  Revised manuscript •  Accepted manuscript •  Version of Record

•  Online publication •  Journal issue

•  Online date •  Issue month

Ø  Adjusting indicators

3 March 2014 15 July 2014

21 January 2015 February 2015

Time Analysis

Time Analysis

Haustein, S., Bowman, T.D. & Costas, R. (in press). When is an article actually published? An analysis of online availability, publication, and indexation dates. Proceedings of the 15th International Conference on Scientometrics and Informetrics, Istanbul, Turkey.

Tweets before publication?

Communities of Attention Opportunity •  Differentiating between types of use •  Measuring social impact

Challenges •  Determining engagement •  Identifying users

Level of engagement on Twitter

Communities of Attention

!Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R. & Larivière, V. (in press), Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter. Journal of the Association for Information Science and Technology. doi: 10.1002/asi.23456

Communities of Attention

Automated Twitter bots

Haustein, S., Bowman, T. D., Holmberg, K., Tsou, A., Sugimoto, C. R. & Larivière, V. (in press), Tweets as impact indicators: Examining the implications of automated “bot” accounts on Twitter. Journal of the Association for Information Science and Technology. doi: 10.1002/asi.23456

tweeting preprints from arXiv subject area

tweeting topic-relevant papers

selection of interesting papers

Communities of Attention

exposure

enga

gem

ent

median dissimilarity with paper title

med

ian

num

ber o

f fol

low

ers

influencers / brokers

orators / discussing

disseminators / mumblers broadcasters

tweet text differs from paper title

tweet text is identical to paper title

few followers many followers Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.

Communities of Attention

708 of 125,083 users (0.6%) tweeting WoS papers published in 2012 (>100) Node size represents number of papers

Network of users tweeting the same papers

Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.

Communities of Attention

Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.

Communities of Attention

Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.

Communities of Attention

Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.

Communities of Attention

Haustein, S., Bowman, T.D. & Costas, R. (in press), ‘Communities of attention’ around scientific publications: who is tweeting about scientific papers? Proceedings of Social Media & Society 2015, Toronto, Canada.

Outlook

Sociological research.

Scientometric studies.

Avoid adverse effects.

Stefanie Haustein

Thank you for your attention! Questions?

stefanie.haustein@umontreal.ca @stefhaustein crc.ebsi.umontreal.ca/sloan

Thank you for your attention!