altmetrics for team science

IUPUI OFFICE OF

ACADEMIC AFFAIRSA

ltm

etri

cs

PRESENTERS:

Heather Coates, Assistant Librarian, University LibraryWillie Miller, Assistant Librarian, University Library

Citation Metrics

Willie Miller

Assistant Librarian, Liaison to the

Schools of Informatics, Journalism,

University Librarian

Altmetrics

Heather Coates

Digital Scholarship & Data

Management Librarian, Liaison to

the Richard M. Fairbanks School

of Public Health, University Library

Traditional Metrics

Team Science & Collaborative Translational Research

March 5, 2015

Willie Miller, MLS Informatics and Journalism Librarian

Traditional Metrics: Levels

• Article-level: citation counts

• Journal-level: impact factor

• Author-level: h-index

Sources of Data

• Web of Science

– Journal Citation Reports (JCR)

• Scopus

• Google Scholar Citation

WEB OF SCIENCE

For author-level, article-level, and journal level metrics (JCR)

SCOPUS

For author-level, article-level, and institution-level metrics

GOOGLE SCHOLAR CITATIONS

For author-level and article-level metrics

Citation Metrics: Tracking

March 5, 2015

Citation Metrics: Tracking

• Team outputs• # of team publications

• Journals for team publications

• High profile citation sources – seminal article, leading researcher in the field, cited in a new policy, etc.

• Personal outputs• Contribution toward key publications or other outputs

• Clearly identify ouputs created as part of a team versus individual or smaller group work

March 5, 2015

Citation Metrics: Presentation

March 5, 2015

5 Top-ranked Journals

JournalNo. of

PublicationsJournal Impact Factor

(2013)5-yr Impact Factor

The Lancet Neurology 3 21.823 24.075

American Journal of Psychiatry 3 13.559 15.062

Brain 3 10.226 10.846

Neurology 11 8.303 8.375

Biological Psychiatry 5 9.472 10.347

Presentation: Examples

March 5, 2015

Team H-index Values

Author H-Index Team Rank

Ira Shoulson 54 1

Elizabeth H. Aylward 53 2

*Jane S. Paulsen 45 3

Julie C. Stout 42 4

Douglas R. Langbehn 33 5

Kevin M. Duff 25 6

AltmetricsTeam Science & Collaborative

Translational Research

March 5, 2015

Heather Coates, MLS, MS Digital Scholarship & Data Management Librarian

Evaluating ImpactTeam Science & Collaborative Translational Research

March 5, 2015

March 5, 2015 https://becker.wustl.edu/impact-assessment

March 5, 2015

Impact Indicators

Research Output& Activities

Biological materials, collaborations, data, databases, repositories, techniques & procedures, grey literature, invention disclosures, mobile apps, patents, trainees, etc.

Advancement of Knowledge

Books/chapters (inclusion in bibliographies, library ownership, textbook use), change in understanding (paradigm shift, lead to new approach), citations (first & second generation citations, countries and institutions represented), conference themes, new centers/institutes

Clinical Implementation(or TRIP)

Biological materials, study cited in clinical decision aid, clinical/practice guidelines, diagnostic application, instruments, quality measure guidelines (gov’t or NPO), reporting requirements

Legislation & Policy

Committee participation, study cited in guidelines, study cited in policy, study cited in enactment of standards

Economic Benefit

Findings cited in reduced costs for delivery of healthcare services, findings result in enhancement of existing resources and expertise, license agreements for use of IP, spinoff or startup company

Community Benefit

Public awareness of risk factors, patient decision materials, cited in public/private insurance benefit plan

AltmetricsTeam Science & Collaborative Translational Research

March 5, 2015

Traditional measures of impact

• Proxy for expert evaluation

• Typically citation-based

• Levels of evidence• Journal-level

• Article-level

• Scholar-level

• How can you use these in your dossier?• What is the value of these metrics?

March 5, 2015

Altmetrics typically measure impact of individual products

Citation-based Metrics

March 5, 2015

Journal Level Metrics

Output/Article Level Metrics

Author Level Metrics

Altmetrics

March 5, 2015

Web MetricsSocial Media

Metrics

View UseShare Recommend Discuss

Altmetric Manifesto

NO ONE CAN READ EVERYTHING. We rely on filters to make sense of the scholarly literature, but the narrow, traditional filters are being swamped. However, the growth of new, online scholarly tools allows us to make new filters; these altmetrics reflect the broad, rapid impact of scholarship in this burgeoning ecosystem. We call for more tools and research based on altmetrics…

March 5, 2015

Altmetrics

• Supplement traditional metrics

• Scope extends beyond the formal published scholarly record (journals & books)

• Timeframe: immediate to short-term impact

• Sources: focus is on social media/engagement

• DOES capture qualitative data (ex: blog snippets, tweet content)

March 5, 2015

Types of scholarly outputs

• Articles

• Books & chapters

• Conference presentations, panels, etc.

• Blogs & blog posts

• Grey literature, conference materials, white papers, unpublished reports

• Learning objects, instructional content, assessment tools, online courses, innovative use of technology, syllabi, etc.

• Impact on diverse populations or communities; impact on the community through media, changes in policy, law, or programs

March 5, 2015

Altmetric Sources

• Publishers (e.g., PLoS, PubMed, etc.)• Aggregation services

– Impact Story: http://impactstory.org/– Altmetric.com: http://www.altmetric.com/– Plum Analytics: https://plu.mx/g/samples

• Subject repositories– PubMedCentral (PMC)– arXiv– SSRN

• Institutional repositories– IUPUI ScholarWorks: http://scholarworks.iupui.edu– IUPUI DataWorks: http://dataworks.iupui.edu

March 5, 2015

Aggregator Sources

• News media

• Social media platforms

• Blogs

• Twitter

• Reference managers (e.g., Mendeley)

• other online conversations captured include YouTube, Wikipedia, Reddit, F1000, Q&A, Policy Docs

• NOT server-side views & download data

March 5, 2015

March 5, 2015

March 5, 2015

March 5, 2015

March 5, 2015

March 5, 2015

March 5, 2015

ImpactStory: Example

March 5, 2015

Plum Analytics Example

March 5, 2015

March 5, 2015

The problem is not a lack of data, but how to evaluate

and make sense of it.

Using Altmetrics

What is your case? What statements do you need to support?

• Identify priorities

• Choose tools/platforms

• Gather

• Select

• Store

• Visualize

March 5, 2015

Gathering Data

• Gather manually– Free

– Time-consuming

– Messy & redundant

– Tailored to your scholarship & argument

– Include unusual or field-specific sources

• Use an aggregator – Minimal cost (individual)

– Less time-consuming

– Generic, broad presentation

– Heavy emphasis on major social media channels

March 5, 2015

Gathering Data: Altmetric.it

• Install Altmetric.it Bookmarklet– http://www.altmetric.com/bookmarklet.php

– Drag the Altmetric.it icon to your browser bookmarks toolbar

• Get altmetrics for your outputs– Go to URL for your article/blog post/column/code/etc

– Click on the Altmetric.it shortcut

– Gather data from the donut in a spreadsheet

• Note: reliant on unique identifiers like DOI, handles, arXiv ID, SSRN ID, PMC ID, etc. but NOT limited to articles

March 5, 2015

Gathering Data: Scopus

• Register for an ORCID (unique author ID)

• Sign up for a Scopus account

• Link your ORCID to your Scopus account

• Maintain an accurate list of articles associated with your profile

• Gather data for each article (track in a spreadsheet)

March 5, 2015

Manual Tracking: Example

March 5, 2015

NOTE: The usage, views, and mentions data are fabricated for this example.

Presentation: Examples

March 5, 2015

Top 5 Articles

Title Year Citations Usage Views Mentions

Apathy is not depression 1998 351 1947 4208 0

*A new model for prediction of the age of onset and penetrance for Huntington's disease based on CAG length 2004 303 2691 5712 2

Generalized cognitive deficits in schizophrenia: A study of first- episode patients 1999 303 1658 3309 0

*Venezuelan kindreds reveal that genetic and environmental factors modulate Huntington's disease age of onset 2004 288 2147 4348 0

*Detection of Huntington's disease decades before diagnosis: The Predict-HD study 2008 234 5819 12381 9

*Product of the Huntington Study Group

NOTE: The usage, views, and mentions data are fabricated for this example.

Presentation: Examples

March 5, 2015

Top 5 Non-Article Outputs

Title Impact Evidence

Huntington Study Group website CommunityReceives more than 2,000 unique visitors per month.

HSG participant database Community 4300 registered potential participants

Research sites Community 63 research sites in 11 countries

Physician's Guide to the Management of Huntington Disease

Clinical Implementation/TRIP Downloaded more than 2,400 times

UHDRSClinical Implementation/TRIP Downloaded more than 13,000 times

NOTE: The evidence details are fabricated for this example.

Sources for Altmetrics by product

• Articles• Publisher website

• PubMedCentral

• SSRN/arXiv/PMC

• IUPUI ScholarWorks

• Presentations• Slideshare

• IUPUI ScholarWorks

• Blogs & blog posts• Wordpress

• Google Analytics

• Twitter

• Data• Figshare

• DataDryad

• Data Journals

• IUPUI DataWorks

March 5, 2015

Using Altmetrics

• Complementary - Portray just part of the picture– More immediate measure of impact

– Measure of impact outside your discipline and academia (public policy, practice, community engagement, public awareness, etc.)

– Reflect engagement with and impact upon a broader audience, beyond academia and your own area of expertise

• Flavors of impact (Priem et al, 2012)– “popular hit” – highly tweeted and shared on non-academic social

media sites

– “expert pick” – good F1000 ratings and subsequent citations, few shares or social media mentions

March 5, 2015

Resources

• ImpactStory Guide to Altmetrics: http://impactstory.demo.libguides.com/c.php?g=211074&p=1392895

• ImpactStory blog (http://blog.impactstory.org/)

• Piwowar, H. & Priem, J. (2013). The power of altmetrics on a CV. Bulletin of the Association for Information Science & Technology. https://www.asis.org/Bulletin/Apr-13/AprMay13_Piwowar_Priem.html

• Roemer, R. C. & Borchardt, R. (2012). From bibliometrics to altmetrics: A changing scholarly landscape. College and Research Libraries News, 73(10), 596-600. http://crln.acrl.org/content/73/10/596.full

• Dinsmore, A., Allen, L., & Dolby, K. (2014). Alternative perspectives on impact: The potential of ALMs and altmetrics to inform funders about research impact. PLoS Biology, 12(11), e1002003. doi: 10.1371/journal.pbio.1002003

• Steven Roberts website & P&T dossier, both incorporating altmetrics: http://faculty.washington.edu/sr320/?p=2806

• Altmetric webinar: Applied Altmetrics: http://godigitalscience.com/view/mail?iID=Y9PXAVDJMH5JVAUPJU79

March 5, 2015

References

• Beacham, B., Kalucy, L., McIntyre, E. (2005). Focus on Understanding and Measuring Research Impact. Retrieved from http://www.phcris.org.au/phplib/filedownload.php?file=/elib/lib/downloaded_files/publications/pdfs/phcris_pub_3236.pdf

• Garfield, E. (2000). The use of JCR and JPI in measuring short and long term journal impact. Presented at the Council of Scientific Editors Annual Meeting: San Diego, CA.

• Leyesdorff, L. (2009). How are new citation-based journal indicators adding to the bibliometric toolbox? Journal of the American Society for Information Science and Technology, 60(7), 1327-1336.

• Priem et al. (2010). Altmetrics Manifesto: http://altmetrics.org/manifesto/

• Priem, J., Piwowar, H., Hemminger, B. M. (2012). Altmetrics in the wild: Using social media to explore scholarly impact. http://arxiv.org/abs/1203.4745

March 5, 2015

QUESTIONS

Concluding Remarks

For additional P&T resources visit

http://academicaffairs.iupui.edu/

P&T Guidelines

Dossier Samples

Online Programs