professional impact of clinical research

Professional impactof clinical research

Gustaf Nelhans

Swedish School of Library and Information Science

STI 2016, Valencia 2016-09-15

Research sponsored by MINSO Solutions, Borås, Sweden

Professional impact

- Academic impact

- Traditional journal based citation analysis measure “inter-academic impact” (disciplinary impact).

- Professional impact:- Publications that have a direct impact on professional practice (e.g. clinical

guidelines, patents, standards)

- Social impact (altmetrics):- E.g. ResearchGate, Academia.edu, Twitter mentions (–relevance for practice?;

academia?)

• This presentation focuses on:

“Professional practice with regards to

clinical practice in the health sector”

Gustaf Nelhans, Swedish School of Library and Information Science, University of Borås, [email protected] 2

Agreement on Medical Education and Research (“ALF-medel”)

• REF based model (SRC 2013)

• Panels evaluating:• Scientific quality: 60 %

• (e.g. bibliometric data, external funding)

• Clinical importance: 30 % • (e.g. self evaluations with case studies)

• Conditions for research 10 % • (e.g.governance, personel with academic degree etc)


Active research on clinical guideline references

• [c]ounting of citations in the serial literature […] may measure academic influence, but the funders of such research are usually more concerned to see if it has had a practical benefit, especially to patients. (Lewison & Sullivan, 2008).

• [n]ational guidelines, those based on systematic reviews and those using explicit evidence-linked methods for deriving recommendations and those being regularly updated and featuring implementation strategies were more likely to be of high quality, than their counterparts. (Andersen, 2013)

• [c]itations from clinical guidelines [...] give evidence that research findings have been used to inform the day-to-day practice of medical staff (Thelwall & Maflahi, 2016)


Citation database for clinical guidelines

Fig. borrowed from Hans Eriksson, Minso ABGustaf Nelhans, Swedish School of Library and Information Science, University of Borås, [email protected] 5

Database and empirical data

Sources No. guidelines

No of refs(with PubMed-id)

SBU 343 27.570

National board of health and welfare (Socialstyrelsen)

49 8.002

- Clin guidelines 13 6.610

• Presently >50 000 references. Of these around 3/4 (72,9 %) are matched to a PubMed-id.

Clinical guidelines TYPE Year RefsStrokesjukvård 2009 CARDIO 2009 645

Vuxentandvård 2011 DENTA 2011 1301

Diabetesvård DIABE 2015 523

Sjukdomsförebyggande metoder GENER 2011 316

Antipsykotisk läkemedelsbehandling vid schizofreni och

schizofreniliknande tillstånd MENTH 2014 25

Demenssjukdom 2010 MENTH 2010 528

Depression och ångestsyndrom 2010 MENTH 2010 567

Missbruk och beroende MENTH 2015 537

Schizofreni eller schizofreniliknande tillstånd 2011 MENTH 2011 106

Bröst-, prostata-, tjocktarms och ändtarmscancervård

2014 ONCOL 2014 1.226

Lungcancervård 2011 ONCOL 2011 277

Palliativ vård i livets slutskede PALLIA 2013 78

Rörelseorganens sjukdomar 2012 REUMA 2012 481

Total 6.610


Descriptive stats of refs (WoS data)

Record type

Article ReviewArticle; Proceedings Paper LetterEditorial Material NoteReview; Book Chapter News Item

Total of 406 references older than 20 years (max= 57)

Mean = 7.8 years

0

100

200

300

400

500

600

700

800

-10 -5 0 5 10 15 20

No

of

refs

Reference age

.

Reference age


Citation data / share of “highly cited”

# Journal C.G. refs Citations Cit/C.G ref

1 JOURNAL OF CLINICAL ONCOLOGY 187 54591 291.9

2 NEW ENGLAND JOURNAL OF MEDICINE 152 167521 1102.1

3 STROKE 144 24435 169.7

4 JOURNAL OF CLINICAL PERIODONTOLOGY 120 7368 61.4

5 COCHRANE DATABASE OF SYSTEMATIC REVIEWS 112 6077 54.3

6 LANCET 109 68433 627.8

7 JAMA-JOURNAL OF THE AMERICAN MEDICAL ASSOCIATION 101 58214 576.4

8 JOURNAL OF PERIODONTOLOGY 81 3373 41.6

9 BRITISH MEDICAL JOURNAL 75 18110 241.5

10 AMERICAN JOURNAL OF PSYCHIATRY 73 11070 151.6

(raw data, not field normalized)


Frequently publishing countries (n=85)

LinLog mode normalizationGustaf Nelhans, Swedish School of Library and Information Science, University of Borås, [email protected] 9

Frequently publishing institutionsOf 6631 institutions, 665 have at least 5 published papers in the set. (Swedish institution names are harmonized)


Gustaf Nelhans, Swedish School of Library and Information Science, University of Borås, [email protected]

Keyword occurrences Keyword occurrences Keyword occurrences

depression 105 prostate cancer 137 meta-analysis 135treatment 71 breast cancer 93 stroke 84randomized controlled trial 51 survival 61 type 2 diabetes 42pain 41 colorectal cancer 53 epidemiology 33alcohol 38 radiotherapy 52 osteoarthritis 26systematic review 106 cost-effectiveness 105 children 22clinical trial 54 dementia 93 cytotoxicity 19periodontitis 30 alzheimer's disease 66 prevalence 14chlorhexidine 29 quality of life 63 allergic contact dermatitis 11dental implants 29 rehabilitation 57 biocompatibility 9

12

Professional Impact: Goals and sources for data

• Clinical guidelines and Treatment recommendations

• …also patents (Wilsdon et al 2015)

• Climate policy documents (Bornmann et al. 2016)

• Standards documents (e.g. ISO, SIS)

• Government White Papers (e.g. Swedish SOU: “Statens Offentliga Utredningar”)

• Basically any issued document having cited references


13

Open questions to relate to if clinical impact for health care should be analysed:

• Professional/clinical impact – theoretic relevance? • ≠ academic impact, should not be treated within a single indicator

• Citation window? • Slightly longer delay before

references are found in clinical guidelines ~8 years:

• How to handle references to non-clinical research?• E.g. references to other clinical guidelines or reviews and not to

underlying clinical research -> follow through iteratively?

• Coverage for source items in PubMed, WoS ~3/4• Is it worth including other sources?

Cited references

Clinical guidelines


14

Conclusions

• In sum, it is argued professional impact, here designated as impact of clinical research within clinical guidelines issued at the national or international level, provides grounds for measuring impact closer to the professional expertise that actually uses the research. Professional impact, as opposed to intra-scientific impact within the academic literature and “social impact”, based on mentions in social media or inclusion in on-line reference databases should be viewed as a distinct instance of research impact that operates on its own specific premises.

Gustaf Nelhans, Swedish School of Library and Information Science, University of Borås,

[email protected]

Thank you! / References

• Andersen, J. P. (2013). Conceptualising research quality in medicine for evaluative bibliometrics. Københavns Universitet: Det Humanistiske Fakultet.

• Bornmann, L., Haunschild, R., & Marx, W. (2015). Policy documents as sources for measuring societal impact: How is climate change research perceived in policy documents? Unpublished manuscript, (arXiv:1512.07071).

• Lewison, G., & Sullivan, R. (2008). The impact of cancer research: how publications influence UK cancer clinical guidelines. British Journal of Cancer, 98(12), 1944-1950.

• Swedish Research Council. (2013). En utvecklad modell för kvalitetsutvärdering av klinisk forskning finansierad av ALF-medel. Stockholm: Swedish Research Council

• Thelwall, M., & Maflahi, N. (2016). Guideline references and academic citations as evidence of the clinical value of health research. Journal of the Association for Information Science and Technology, 67(4), 960-966.

• Wilsdon, J., et al.,. (2015). The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management.


16

professional impact of clinical research

Health & Medicine