practical applications for altmetrics in a changing metrics landscape
TRANSCRIPT
Practical applications for altmetrics in a changing
metrics landscape
Sara Rouhi, @RouhiRooProduct Specialist, Altmetric
Anirvan Chatterjee, @anirvanDirector, Data Strategy
Clinical & Translational Science Institute, [email protected]
Today we’ll cover
• Need and origins• Definitions• Where you can find altmetrics• How they’re being used at UCSF• How they’re being used at Duke
University• What the future may bring…
Like all great movements…
It started with a manifesto…www.altmetrics.org
If metrics are about filtering the good research from the bad, traditional metrics* aren’t working
*Peer review, journal impact factor, citation counting
*http://blogs.nature.com/news/2014/05/global-scientific-output-doubles-every-nine-years.html
Global scientific output doubles every nine years
From submission to publication in as little as six weeks
Traditional metrics lag behind…
44K
online mentions of scholarly articles every day.
1 mention every 2 seconds!
50K unique articles are shared each week.
>3.5M
articles with tracked attention data.
The conversation has moved online..
Source: Altmetric internal data, March 2015
Funders want evidence of societal impact
Grant funders looking for proof of “broader impacts” often defined as “an effect, change, or benefit to the economy, society, culture, public policies, health, the environment, etc.”
Research Excellence Framework, http://www.ref.ac.uk/panels/assessmentcriteriaandleveldefinitions/
Broaden dissemination to enhance scientific and technological understanding, for example, by presenting results of research and education projects in formats useful to students, scientists and engineers, members of Congress, teachers, and the general public.
http://www.nsf.gov/pubs/2007/nsf07046/nsf07046.jsp
Expanded administrative remits
• Strategic planning• Supervision of academic affairs • Fundraising• Grants administration • Public affairs
So what is Altmetric and what are altmetrics?
Altmetric is a data science company that tracks attention to research outputs, delivering output level metrics via visually engaging, intuitive interfaces.
In other words, we help give credit where credit is due.
Who are we?
An alternative,
more immediate measure of attention
From non-traditional
sources
To provide a larger context
What are altmetrics?
Not a replacement but a complement
Policy documents, blogs, mainstream news, social media
Providing a multi-faceted picture of engagement
Multifaceted picture of engagement: Audiences
Practitioners General Public
Professional Communicators
Interested Parties
Scholars
Multi-faceted picture of engagement: Interaction
• Scholars– Downloads – Citations– Bookmarks/saves
• Early career– Social media, blogs
• General public– News, blogs, – Social media
• Practitioners– Policy documents– Field-specific blogs/Social
Media
• Research communicators– News, blogs, social media
• Interested parties– Policy docs, blogs
Who on campus needs to track this?
Administrators(Grants,
Departmental, Institutional)
Library
Marketing/PR/Communication
s
Research Groups
Why? Administrators
• Are we in compliance with grant/govt mandates?
• Do our research outputs work toward our group/dept/instit. mission?
• Does our campus have global reach?
• Does our research influence policy, legislation, best practices?
Why? Libraries
• Do our collections reflect where our research gets the most attention (i.e. are we missing anything? Are we purchasing the wrong things?)
• Does our OA policy bring more attention to our work?
• How does our institutional repository bring attention to campus research?
Why? Marketing/PR/Communications
• Is anyone out there getting it wrong?• Have we missed opportunities to get in
front of a PR/communications storm?• Can we benchmark our outreach
efforts?• Are we reaching the target markets we
want?• Are we using the right media?
Why? Research Groups
• Are we reaching the audiences we want to see our work?
• Is anyone misrepresenting/confused by our work?
• How do we demonstrate “broader impact” to grant funders?
• How can we reach more people with our research?
• Are we engaging unexpected communities?
So where will you find altmetrics?
Where will you see our data? Publisher platforms
Where will you see our data? Books
Where will you see our data? Other metrics providers
Where will you see our data? Author Tools
“A CV that documents alternative metrics […] offers a much more compelling argument to a tenure committee of their research impact than a traditional publication list.”
- Donald Samulack, Editage
Recommendation Engine Integration for Medical research Apps
Integrating Altmetric service into publishing platform
Altmetric Integration for JAMA and others to monitor research impact
Integrates Altmetric data for over 1 million articles
Where will you see our data? Platforms
• Institutional repository badge embeds
• Badge integration with discovery systems
Where will you see our data? Institutional repositories/discovery systems
Who uses our data? Research Management Systems
In practice, how do you use this data?
Three Experimentswith Altmetric data
April 22, 2015
Anirvan ChatterjeeDirector, Data StrategyClinical & Translational Science Institute
Clinical & TranslationalScience Institute
UCSF Profiles profiles.ucsf.edu
Research networking system (like VIVO, Symplectic Elements)
Research profile of 7,000 people on campus
• Bios, publications, NIH grants, awards, etc.
Not just a directory
Publications automatically kept current
• Heavily used — 100,000+ visits per month from on/off campus
• Data reuse — APIs used by 25 other campus systems
Why altmetrics?
Show early impacts of research
Attempt to measure/visualize impact, rather than just anecdata
Doesn’t displace traditional metrics of research output(e.g. citations, journal rankings, etc.)
build quick • fail fast
build quick • fail fast
three experiments
1. Department Data Explorer
Lessons learned
People were interested in seeing what new research was trending
Altmetric badges were easy to integrate
2. UCSF Profiles publication list
Lessons learned
Altmetrics advocates were supportive
Zero pushback from campus community
Because of easy Altmetric integration, we could addaltmetrics even before we added citation data
3. Article recommendation engine
Background
Many researchers focus on a handful of key journals, but may miss out on trending stories on non-core topics of interest
• e.g. cardiologist interested in digital health
We know UCSF researchers’ research topics/interests…
• Hand-entered
• Algorithmically derived from publications
Our recommendation engine shares new articles of interest that matches researchers’ known areas of interest
Altmetric API
Details at http://api.altmetric.com/
Free to use basic data for apps and mashups, with rate limits
Generous free access for noncommercial academic research projects
04/18/2023Presentation Title and/or Sub Brand Name Here55
Lessons learned so far (work in progress) Altmetric API made it easy to integrate altmetrics data
Among first round of beta testers:
• Most hadn’t seen recommended papers
• Some questions about article level metrics vs. journal reputation
• Need to improve relevance matching
• Enough positive feedback for us to keep exploring
Takeaways from our three experiments… When it comes to altmetrics, researchers aren’t monolithic
• Some bullish, others guardedly positive, few/none offended
Altmetrics data doesn’t yet solve a burning institutional need
• We’re hearing more about altmetrics from early adopters, rather than leadership
Low barriers to experimentation
• It’s very easy to get started and integrate into our processes
• We’re able to keep tossing around ideas to find the best fit
Anirvan ChatterjeeClinical & Translational Science Institute, [email protected] • @anirvan
NSF Broader Impacts Criterion
To what extent will [the research] enhance the infrastructure for research and education, such as facilities, instrumentation, networks, and partnerships?
Will the results be disseminated broadly to enhance scientific and technological understanding?
What may be the benefits of the proposed activity to society?
http://www.nsf.gov/pubs/2007/nsf07046/nsf07046.jsp NSF Broader Impacts Criterion
Have you heard of…
“Try the bookmarklet, it’s ultra cool”
Article by article wasn’t enough
My data from the Altmetric for Institutions summary report
Policy documents? Who knew?
Demonstrating “broader impact” with International News coverage = 60%
Data from Article Details Pages
Demonstrating “broader impact” with International Blog coverage = 40%
Data from Article Details Pages
New communities, new conversations
From Article Details Pages
Many many more eyeballs
Article 1 Article 2 Article 3 Article 4 Article 5 Article 6 Article 7 Article 8 Article 9 Article 100
500,000
1,000,000
1,500,000
2,000,000
2,500,000
3,000,000
3,500,000
4,000,000
4,500,000
3,941,227
634,343
190,593 187,480 263,71970,617 137,926 115,455 84,158 5,121
Twitter reach by article - Total upward bound: 5,630,639
Data from Article Details Pages
Even if 1% click on the article, that’s 56,000 eyes that never would have seen it before Twitter.
Saved Terrie time; saved her program manager time…
NIH Program Manager:“[This Altmetric data
is] fantastic information for [our]
budget report.”
Before Altmetric data she didn’t know…
• How broadly her work was disseminated – News vs policy vs blogosphere
• The difference in interest by source– Methodology papers via Twitter
• That all this data could be aggregated to save time
Recap of where we are…
• Education is critical• Tenure/promotion paradigm• “Here one day, gone the next”• Need for sentiment analysis
– So it’s not just more numbers
• Facilitating industry standards– NISO Altmetrics Whitepaper
Attention exists on a spectrum
Tweets/bookmark
s
Holdings/saves/shares
Usage CitationsPolicy
document citations
Blog coverage
Post publicatio
n peer review
News coverage
• Superficial • Article may or may not have been
read• Many potential readers but few
actual• Cost-light (er)
• Article more likely to be read• Cost-heavy (ier)• Readers = practitioners (?)• Actionable (?)