impact assessment in the funding sector: the role of altmetrics

24
Impact assessment in the funding sector: the role of altmetrics Adam Dinsmore [email protected]

Upload: aileen-ramirez

Post on 03-Jan-2016

20 views

Category:

Documents


1 download

DESCRIPTION

Impact assessment in the funding sector: the role of altmetrics. Adam Dinsmore [email protected]. Outline. The Wellcome Trust Altmetrics Why they are important to us How we currently use them The altmetrics ecosystem Consistency, transparency, availability. The Wellcome Trust. - PowerPoint PPT Presentation

TRANSCRIPT

Page 1: Impact assessment in the funding sector: the role of altmetrics

Impact assessment in the funding sector: the role of altmetrics

Adam [email protected]

Page 2: Impact assessment in the funding sector: the role of altmetrics

The Wellcome Trust

Altmetrics Why they are important to us How we currently use them

The altmetrics ecosystem Consistency, transparency, availability

Outline

Page 3: Impact assessment in the funding sector: the role of altmetrics

Set up in 1936 under the will of Sir Henry Wellcome.

Our vision is to achieve extraordinary improvements in human and animal health.

Our mission is to support the brightest minds in biomedical research and the medical humanities.

We spent approx £538 million on research in FY 2012/13.

The Wellcome Trust

Page 4: Impact assessment in the funding sector: the role of altmetrics

Current grant portfolio

Page 6: Impact assessment in the funding sector: the role of altmetrics

Monitoring progress: WT’s key indicatorsOutcomes Key indicators of progress

Discoveries

Applications

Engagement

Research leaders

Research environment

Influence

1. significant advances in the generation of new knowledge2. contribute to discoveries with tangible impacts on health

3. contribute to the development of enabling technologies, products and devices4. uptake of research into policy and practice

5. enhanced level of informed debate in biomedicine6. significant engagement of key audiences & increased reach

7. develop a cadre of research leaders8. evidence of significant career progression among those we support

9. key contributions to the creation, development and maintenance of major research resources

10. contributions to the growth of centres of excellence

11. significant impact on science funding & policy developments12. significant impact on global research priorities and processes

Page 7: Impact assessment in the funding sector: the role of altmetrics

The Wellcome Trust: OA Policy Supports unrestricted access to the published output of

research as a fundamental part of its charitable mission. Expects authors to maximise the opportunities to make their

results available for free. Requires papers be made available through PubMed Central

within six months of publication. Provides additional funding to cover open access charges. Encourages Creative Commons Attribution licence (CC-BY).

Affirms the principle that it is the intrinsic merit of the work, and not the title of the journal in which the author’s work is published, that should be considered in making funding decisions.

Page 8: Impact assessment in the funding sector: the role of altmetrics
Page 9: Impact assessment in the funding sector: the role of altmetrics

o Cited 2904 times; o Normalised Citation

Impact = 327;

o Acta Crystal D JIF = 7.232

Article level metrics vs. Journal level metrics

Page 10: Impact assessment in the funding sector: the role of altmetrics

Current use of altmetrics at WellcomeEngagement/Influence beyond citations

Page 11: Impact assessment in the funding sector: the role of altmetrics

MEP

Centre for Bioethics

MEP

Professor of EBM

Journal editor

Health journalist

NGO

Health, Population & Nutrition @ The World

Bank

Engagement/Influence beyond citations

Dinsmore, Adam
Of these twitter accounts, only the attention of the Centre for Bioethics and possibly the journal editor are likely to be reflected in ciation metrics (eventually).
Dinsmore, Adam
Be ggod to say something about our next job as 'the Evaluation Team' in a funder - having presented such examples - would be to track whether any policy changes were effected following this publication and tweet etc. The challenges for us (as ever!) is to work through the policy making process to help research evidence get into policy and practice - this has always been a challenge but now we have a greater opportunity (and potentially quicker route - via open access and social media etc) for research findings to reach those we seek to influence ... And this kind of policy impact migh be lots quicker than it used to be -
Page 12: Impact assessment in the funding sector: the role of altmetrics

• Paper cited 9 times to year end 2012.

• However, tweeted about 164 times; among the highest ever for Nat. Neuroscience.

• Identity of tweeters possibly indicative of impact on policy, public debate.

Science journalist, Author

Editor, Medical History

Intelligence Consultant

Engagement/Influence beyond citations

Page 13: Impact assessment in the funding sector: the role of altmetrics

• 2012 Trust-associated paper in PLOS Neglected Tropical Diseases.

• Cited very few times, but discussed extensively on social media.

• However, likely due to funny title rather than genuine public engagement.

Page 14: Impact assessment in the funding sector: the role of altmetrics

Monitoring progress: WT’s key indicatorsOutcomes Key indicators of progress

Discoveries

Applications

Engagement

Research leaders

Research environment

Influence

1. significant advances in the generation of new knowledge2. contribute to discoveries with tangible impacts on health

3. contribute to the development of enabling technologies, products and devices4. uptake of research into policy and practice

5. enhanced level of informed debate in biomedicine6. significant engagement of key audiences & increased reach

7. develop a cadre of research leaders8. evidence of significant career progression among those we support

9. key contributions to the creation, development and maintenance of major research resources

10. contributions to the growth of centres of excellence

11. significant impact on science funding & policy developments12. significant impact on global research priorities and processes

Page 15: Impact assessment in the funding sector: the role of altmetrics

Monitoring progress: WT’s key indicatorsOutcomes Key indicators of progress

Discoveries

Applications

Engagement

Research leaders

Research environment

Influence

1. significant advances in the generation of new knowledge2. contribute to discoveries with tangible impacts on health

3. contribute to the development of enabling technologies, products and devices4. uptake of research into policy and practice

5. enhanced level of informed debate in biomedicine6. significant engagement of key audiences & increased reach

7. develop a cadre of research leaders8. evidence of significant career progression among those we support

9. key contributions to the creation, development and maintenance of major research resources

10. contributions to the growth of centres of excellence

11. significant impact on science funding & policy developments12. significant impact on global research priorities and processes

Page 16: Impact assessment in the funding sector: the role of altmetrics

What we’re currently exploring…

Cell, Developmental &

Phys Scie

nces (

n=500)

Genetic & M

olecular S

cience

s

(n=622)

Infection and Im

muno-Biology (n=1072)

Medical H

umanities (n=116)

Molecules, G

enes and Cells

(n=194)

Neurosci

ence and M

ental Health

(n=994)

Physiologica

l Scie

nces

(n=66)

Population Health

(n=748)

00.20.40.60.8

11.21.4

PDF citations of selected Wellcome Trust associated research(2011-2012)

(.ac.uk, .edu, .gov, .gov.uk, .org.uk, academic syllabi)

Funding streamBase: 4,312 Wellcome Trust associated papers

Mea

n PD

F ci

tatio

ns

Credit: Statistical Cybermetrics Research Group, University of Wolverhampton.

Page 17: Impact assessment in the funding sector: the role of altmetrics

Cell, Developmental &

Phys Scie

nces (

n=500)

Genetic & M

olecular S

cience

s

(n=622)

Infection and Im

muno-Biology (n=1072)

Medical H

umanities (n=116)

Molecules, G

enes and Cells

(n=194)

Neurosci

ence and M

ental Health

(n=994)

Physiologica

l Scie

nces

(n=66)

Population Health

(n=748)

00.005

0.010.015

0.020.025

0.030.035

0.040.045

Academic syllabi citations of selected Wellcome Trust associated research (2011-12)

Funding streamBase: 4,312 Wellcome Trust associated papers

Mea

n ac

adem

ic sy

llabi

cita

tions

What we’re currently exploring…

Credit: Statistical Cybermetrics Research Group, University of Wolverhampton.

Page 18: Impact assessment in the funding sector: the role of altmetrics

What we’re currently exploring…

Page 19: Impact assessment in the funding sector: the role of altmetrics

Altmetrics ecosystem

Consistency (of definitions)

Transparency

Availability

Page 20: Impact assessment in the funding sector: the role of altmetrics

Altmetrics ecosystem - Sources of data We need metrics data which is consistent in its source and

meaning in order to enable sensible comparisons between outputs of different schemes

Differences between publisher-provided data mean that, primarily, we use third-party data providers:

Thomson Reuters Web of Knowledge Altmetric

Page 21: Impact assessment in the funding sector: the role of altmetrics

Differences in citation data

Page 22: Impact assessment in the funding sector: the role of altmetrics

Source Number of citations

Scopus 76Web of Science 64Google Scholar 103CrossRef 56PMC 44EuropePMC 66

For a sample of 358 Wellcome-associated papers, on average: WoS had 10 cites per paper Scopus 12 cites per paper Google Scholar 18 cites per paper

Differences in citation data

Page 23: Impact assessment in the funding sector: the role of altmetrics

Summary:

The Wellcome Trust is interested in all of the different types of impact which arise from the work it supports (and numerous means of measuring it).

Therefore delighted that helpful tools are being developed.

Greater consistency, transparency, and availability would serve the Trust’s requirements.

Page 24: Impact assessment in the funding sector: the role of altmetrics

-Thank You-Adam DinsmoreWellcome Trust

Strategic Planning & Policy Unit

[email protected]