niso webinar: new perspectives on assessment how altmetrics measure scholarly impact
TRANSCRIPT
NISO Webinar: New Perspectives on Assessment How Altmetrics Measure Scholarly Impact
November 13, 2013
Speakers:
Euan Adie - Founder, altmetric.com Stefanie Haustein, Ph.D. - Research Analyst at Science-Metrix
Mike Taylor - Research Specialist, Elsevier Labs
http://www.niso.org/news/events/2013/webinars/altmetrics
Beyond traditional impact: what can altmetrics do for you?
Euan Adie, altmetric.com NISO webinar, 13th November 2013
Several different tools available
What are “altmetrics”?
o “alternative metrics”
o new ways of measuring different, non-traditional forms of impact, potentially of non-traditional outputs.
o “alternative to only using citations”, not “alternative to citations”.
o complementary to traditional citation-based analysis.
Every researcher is a communicator
Within academia Presentations and seminars
Funding and ethics applications Academic books
Journal articles and posters Term papers and essays
Meetings and conferences Correspondence
Within society Speaking at public events Books for general audiences Press Social media Blogs Policy documents
We should measure both
Download counts
Page views Mentions in news reports Mentions in social media
Mentions in blogs Reference manager readers
… etc.
Journal Impact Factor
Citation counts
New perspectives of impact
ACADEMIC IMPACT SOCIETAL IMPACT
Alternative metrics “altmetrics”
+ Traditional metrics Traditional metrics
One available tool. We are used a lot by publishers, now some institutions too. We serve ± 2.5 million requests a day.
Where are the readers?
Who are the readers?
What’s interes3ng about this kind of data?
Altmetrics tools are at version 0.1
Altmetrics tools don’t (yet) provide good metrics for
impact
BUT
They can help you find evidence of impact, successes
An article on the ecological impacts of the Fukushima nuclear accident.
• > 1,859 twitter accounts shared, combined follower count of 2.5M.
• 68% of tweets sent from Japan.
• 77% of tweets from members
of the public.
2012, Scientific Reports 2, 570 Figure 1
Evidence of public outreach?
A different example, from the USP
Evidence research has reached patients?
PLOS ALM Reports: An exploratory review Engagement/Influence beyond citaEons
q No cita3ons in 3 months since publica3on.
q However TwiJer men3ons 70x the average ar3cle in our dataset.
q 4x the average for PLOS Medicine ar3cles in 2013.
PLOS ALM Reports: An exploratory review Engagement/Influence beyond citaEons
MEP
Centre for Bioethics
MEP
Professor of EBM
Journal editor
Health journalist
NGO
Health, PopulaEon & NutriEon @ The World
Bank
Engagement/Influence beyond citaEons
Monitoring progress: WT’s key indicators
Outcomes Key indicators of progress
Discoveries
ApplicaEons Engagement
Research leaders Research environment
Influence
1. significant advances in the genera3on of new knowledge 2. contribute to discoveries with tangible impacts on health
3. contribute to the development of enabling technologies, products and devices
4. uptake of research into policy and prac3ce
5. enhanced level of informed debate in biomedicine 6. significant engagement of key audiences & increased reach
7. develop a cadre of research leaders 8. evidence of significant career progression among those we support
9. key contribu3ons to the crea3on, development and maintenance of major research resources
10. contribu3ons to the growth of centres of excellence
11. significant impact on science funding & policy developments 12. significant impact on global research priori3es and processes
Altmetric score
Quan3fying aJen3on
Note that we can measure aJen3on, but…
Posi3ve? Nega3ve?
For scien3fic reasons? Or because the 3tle is funny? Is 364 good or bad anyway?
Context is everything
In general, altmetrics numbers…
X Don’t represent the quality of research.
Don’t indicate the quality of individual researchers.
Don’t tell the whole story – always look for qualitative data as well
X
X
Why score at all? To allow ranking
What should we be measuring beyond aJen3on?
Ques3on for the academic community.
Problems
Problems
• 30 – 40% of recent biomedical papers will have Altmetric aJen3on. But < 10% in social sciences.
Problems
• 30 – 40% of recent biomedical papers will have Altmetric aJen3on. But < 10% in social sciences.
• Tools have subtle bias: data sources are mainly those popular in US, Europe
hJp://am.ascb.org/dora/
Thanks for listening!
Supported by:
E-mail: [email protected] Twitter: @altmetric Website: altmetric.com
Disciplinary differences and other biases Exploring social media metrics in scholarly context
[email protected] @stefhaustein Stefanie Haustein
Overview • Altmetrics: definitions • Bibliometrics: in retrospect • Altmetrics: present
• correlations • publication age biases • disciplinary biases • subject biases
• Altmetrics: future • References
Altmetrics: definitions • term coined by Jason Priem • introduced as a better filter
than and alternative to citations and peer-review
http://altmetrics.org/manifesto/ • “…altmetrics is a good idea,
but a bad name” “…we would like to propose the term influmetrics”
Rousseau & Ye (2013)
• rather complementary than alternative to citations
• social media metrics
Altmetrics: definitions • ultimate goals
• similar to but more timely than citations Ø predicting scientific impact
• different, broader impact than captured by citations Ø measuring societal impact
• impact of various outputs Ø “value all research products”
Piwowar (2013)
Altmetrics: definitions • Altmetrics are “representing very different things”
(Lin & Fenner, 2013)
• unclear what exactly they measure: • scientific impact • social impact • “buzz” • all of the above?
Altmetrics: definitions
ad-hoc classifications need to be supported by research
Altmetrics: definitions scientist on Twitter tweeting scientific paper in non-scholarly manner: • scientific impact? • social impact? • buzz?
Altmetrics: definitions • complex to define and classify tools and motivations
• scientific and non-scientific audiences cannot be determined on the platform used
• level of engagement differs not only between platforms but also within:
saving paper to Mendeley library vs. tweeting about it saving vs. reading
retweeting link vs. discussing content
Ø differentiation between audiences and engagement needed to determine meaning of metrics
Bibliometrics: in retrospect • when Garfield created SCI, sociologists of science
analyzed meaning of publications and citations (Merton, Zuckerman, Cole & Cole, etc.)
• sociological research • What is it to publish a paper? • What are the reasons to cite?
• empirical bibliometric research • disciplinary differences in publication
and citation behavior • delay and obsolescence patterns
Bibliometrics: in retrospect • empirical studies helped sociologists to understand
structure and norms of science • for bibliometricians, studies provided a theoretical
framework and legitimation to use citation analysis in research evaluation
• knowledge about disciplinary differences and obsolescence patterns helped to normalize statistics and create more appropriate indicators
Bibliometrics: in retrospect • similar to development of SCI in the 1960s, social
media metrics have to be analyzed: • qualitative studies to analyze who, how and why
people use various social media platforms • large-scale quantitative studies to determine
differences and biases in terms of disciplines, topics, document types, publications years, publication types and sources, author age and affiliation, etc.
Ø to find out what various social media metrics mean and what they can be used for
Altmetrics: correlations e.g., Mendeley • 793 Nature papers: ρ=0.559
820 Science papers: ρ=0.540 • 1,651 JASIST papers: ρ=0.458 • 5,596 PLoS ONE papers: ρ=0.3 • 1,136 bibliometrics papers: ρ=0.448 • 1,389 F1000 papers: ρ=0.686 • 62,647 social science papers: ρ=0.516
14,640 humanities papers: ρ=0.428 • random sample of
200,000 WoS papers: ρ=0.35 • 586,600 PubMed papers: ρ=0.386
Bar-Ilan (2012)
Priem, Piwowar, & Hemminger (2012)
Bar-Ilan et al. (2011)
Li, & Thelwall (2012)
Mohammadi & Thelwall (in press)
Zahedi, Costas, & Wouters (2013)
Haustein, et al.(submitted)
Li, Thelwall, & Giustini (2012)
Altmetrics: age biases Current biases influencing correlation coefficients
Altmetrics: age biases Current biases influencing correlation coefficients
Altmetrics: age biases Current biases influencing correlation coefficients
Altmetrics: age biases Current biases influencing correlation coefficients
Ø compare documents of similar age Ø normalize for age differences
Altmetrics: disciplinary biases PubMed papers covered by Web of Science 2010-2012
Altmetrics: disciplinary biases PubMed papers covered by Web of Science 2010-2012
Altmetrics: disciplinary biases x-axis: coverage of specialty on platform y-axis: correlation between social media counts and citations bubble size: intensity of use based on mean social media count rate
Altmetrics: subject bias
Scatterplot of number of citations and number of tweets (A, ρ=0.181**) and Mendeley readers (B, ρ=0.677**), bubble size represents number of Mendeley readers (A) and tweets (B). The respective three most tweeted (A) and read (B) papers are labeled showing the first author.
General Biomedical Research papers 2011
Altmetrics: subject bias
Article Journal C T
Hess et al. (2011). Gain of chromosome band 7q11 in papillary thyroid carcinomas of young patients is associated with exposure to low-dose irradiation PNAS 9 963
Yasunari et al. (2011). Cesium-137 deposition and contamination of Japanese soils due to the Fukushima nuclear accident PNAS 30 639
Sparrow et al. (2011). Google Effects on Memory: Cognitive Consequences of Having Information at Our Fingertips Science 11 558
Onuma et al. (2011). Rebirth of a Dead Belousov–Zhabotinsky Oscillator Journal of Physical Chemistry A -- 549
Silverberg (2012). Whey protein precipitating moderate to severe acne flares in 5 teenaged athletes Cutis -- 477
Wen et al. (2011). Minimum amount of physical activity for reduced mortality and extended life expectancy: a prospective cohort study Lancet 51 419
Kramer (2011). Penile Fracture Seems More Likely During Sex Under Stressful Situations Journal of Sexual Medicine -- 392
Newman & Feldman (2011). Copyright and Open Access at the Bedside New England Journal of Medicine 3 332
Reaves et al. (2012). Absence of Detectable Arsenate in DNA from Arsenate-Grown GFAJ-1 Cells Science 5 323
Bravo et al. (2011). Ingestion of Lactobacillus strain regulates emotional behavior and central GABA receptor expression in a mouse via the vagus nerve PNAS 31 297
Top 10 tweeted documents: catastrophe & topical / web & social media / curious story scientific discovery / health implication / scholarly community
Altmetrics: future • before applying social media counts in information
retrieval and research evaluation, we need: Ø to understand and define meaning of various
social media metrics Ø to identify different biases Ø to differentiate between audiences and
level of engagement Ø more transparency and reliability in data aggregation
References Bar-Ilan, J. (2011). Articles tagged by 'bibliometrics' on Mendeley and CiteULike. Paper presented at the Metrics 2011 Symposium on Informetric and Scientometric Research, New Orleans, Louisiana. Bar-Ilan, J., Haustein, S., Peters, I., Priem, J., Shema, H., & Terliesner, J. (2012). Beyond citations: Scholars' visibility on the social web. In Proceedings of the 17th International Conference on Science and Technology Indicators, (pp. 98-109). Haustein, S., Peters, I., Sugimoto, C.R., Thelwall, M., & Larivière, V. (in press). Tweeting biomedicine: an analysis of tweets and citations in the biomedical literature. Journal of the American Society for Information Science and Technology. Haustein, S., Bowman, T.D., Holmberg, K., Larivière, V., & Peters, I., (submitted). Astrophysicists on Twitter: An in-depth analysis of tweeting and scientific publication behavior. Aslib Proceedings. jasonpriem (2010, September 28). I like the term #articlelevelmetrics, but it fails to imply *diversity* of measures. Lately, I'm liking #altmetrics. [Twitter post]. Li, X. & Thelwall, M. (2012). F1000, Mendeley and Traditional Bibliometric Indicators. In Proceedings of the 17th International Conference on Science and Technology Indicators, (pp. 541-551). Li, X., Thelwall, M., & Giustini, D. (2012). Validating online reference managers for scholarly impact measurement. Scientometrics, 91(2), 461-471. Lin, J. & Fenner, M. (2013). Altmetrics in evolution: Defining and redefining the ontology of article-level metrics. Information Standards Quarterly, 25(2), 20-26. Mohammadi, E., & Thelwall, M. (in press). Mendeley readership altmetrics for the social sciences and humanities: Research evaluation and knowledge flows. Journal of the American Society for Information Science and Technology. Piwowar, H. (2013). Value all research products. Nature, 493(7431), 159. Priem, J., Piwowar, H., & Hemminger, B.M. (2012). Altmetrics in the wild: Using social media to explore scholarly impact. arXiv. Priem, J., Taraborelli, D., Groth, P. & Neylon, C. (2010). Alt-Metrics: A Manifesto. Rousseau, R., & Ye, F.Y. (2013). A multi-metric approach for research evaluation. Chinese Science Bulletin, 58(26), 3288-3290. Zahedi, Z., Costas, R., & Wouters, P. (2013). What is the impact of the publications read by the different Mendeley users? Could they help to identify alternative types of impact? Presentation held at the PLoS ALM Workshop 2013 in San Francisco.
Scholarly connecEons Cita3ons, social media, ORCID and authorship networks
Mike Taylor Research Specialist
hJp://orcid.org/0000-‐0002-‐8534-‐5985 @herrison
The number of possible connec3ons between researchers and ar3cles, researchers and researchers, and ar3cles and ar3cles is accelera3ng drama3cally. Although bibliometrics has been studied for 50 years, the study of these new connec3ons has only been undertaken recently. As infrastructure is built to accommodate this massively connected world, so research becomes enabled and desirable. Part 1 – formal links Part 2 – informal, ad hoc links
A person writes an ar3cle
A person reads & cites other ar3cles
Ad nauseum
A person writes an ar3cle with another person
Not everyone did enough to be an ‘author’
Some3mes people in the same field have the same name
Some3mes people with the same name write the same paper
?
Some3mes people with the same name get credit for papers they didn’t write
!
End of part 1 Known facts: A person writes an ar3cle A person reads & cites other ar3cles A person writes an ar3cle with another person -‐ Bibliometrics QuesEons about facts: Not everyone did enough to be an ‘author’ -‐ Ethics, acknowledgement statements, contributorship Problems about facts: Some3mes people in the same field have the same name Some3mes people with the same name write the same paper Some3mes people with the same name get credit for papers they didn’t write -‐ ORCID, over 300,000 ORCIDs aler a year, eg, Elsevier's editorial system has
over 100,000 ar3cles in produc3on with ORCIDs
A person cites an ar3cle
A person does X with an ar3cle
pins
Re-‐uses
Writes a blog
Saves on delicious
tweets
Facebooks Saves on Mendeley Writes a newspaper
ar3cle
Different kinds of outputs pins
Re-‐uses
Writes a blog
Saves on delicious
tweets Facebooks
Saves on Zotero / Mendeley / Citeulike / biblio tool
Writes a newspaper ar3cle
Social network ac3vity pins
Saves on delicious
tweets Facebooks
Re-‐using data, graphics, code
Re-‐uses
Scholarly sharing / bookmarking / recommenda3ons
Saves on Zotero / Mendeley / Citeulike / biblio tool
Document crea3on
Writes a blog
Writes a newspaper ar3cle
End of part two Known facts: There are more connec3ons now than have ever been Of more types than ever Crea3on is ad hoc, post hoc, technocra3c, automa3c, pragma3c, real-‐3me… We can count things we don’t understand Emergent thoughts: An ORCID can be seen as a document about a person Links between documents can be formed with no human cura3on (seman3c web) Altmetrics gives us a view into the world of connecEons, as a very limited starEng point: We need research into meaning and correla3on before we can make conclusions – researchers Issues of iden3ty, privacy, seman3cs, authorship / contributorship, cura3on are all in3mately bound with altmetrics
Appendix: seven use cases for altmetrics 1. Predic3on of ul3mate cita3on 2. Measuring / recognizing component re-‐use /
preparatory work / reproducibility 3. Hidden impact (impact without cita3on) 4. Real-‐3me filtering / real-‐3me evalua3on (sigint) 5. Plaporm / publisher / ins3tu3on comparison 6. Measuring social reach / es3ma3ng social impact 7. Altmetrics is of interest by itself
NISO Webinar • November 13, 2013
!!!!!!!!!!!!!!!!!!Questions?!All questions will be posted with presenter answers on the NISO website following the webinar:!!http://www.niso.org/news/events/2013/webinars/altmetrics
NISO Webinar: New Perspectives on Assessment How Altmetrics Measure Scholarly Impact
Thank you for joining us today.
Please take a moment to fill out the brief online survey.
We look forward to hearing from you!
THANK YOU