the pernicious habit of ranking scientists by the journals they publish in
DESCRIPTION
The empirical evidence against the use of journal rank as an evaluation tool and how to fix the scientific infrastructure.TRANSCRIPT
![Page 1: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/1.jpg)
Deep Impact: The pernicious habit of ranking scientists by the
journals they publish in
Björn BrembsUniversität Regensburg
http://brembs.net
![Page 2: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/2.jpg)
More scientists, more publications
![Page 3: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/3.jpg)
Journal Rank
• Thomson Reuters: Impact Factor• Eigenfactor (now Thomson Reuters)• ScImago JournalRank (SJR)• Scopus: SNIP, SJR
Source Normalized Impact per Paper
![Page 4: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/4.jpg)
Journal Rank
Only read publications from high-ranking journals
![Page 5: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/5.jpg)
Job applications
![Page 6: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/6.jpg)
Job application instructions
Publikationstätigkeit(vollständige Publikationsliste, darunter Originalarbeiten als Erstautor/in, Seniorautor/in, Impact-Punkte insgesamt und in den letzten 5 Jahren, darunter jeweils gesondert ausgewiesen als Erst- und Seniorautor/in, persönlicher Scientific Citations Index (SCI, h-Index nach Web of Science) über alle Arbeiten)
Publications:Complete list of publications, including original research papers as first author, senior author, impact points total and in the last 5 years, with marked first and last-authorships, personal Scientific Citations Index (SCI, h-Index according to Web of Science) for all publications.
![Page 7: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/7.jpg)
Journal Rank
Only read publications from high-ranking journals
![Page 8: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/8.jpg)
Journal Rank
Only publish in high-ranking journals
![Page 9: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/9.jpg)
METRICS
Is journal rank like astrology?
![Page 10: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/10.jpg)
Show of hands:
• Who knows what the IF is?• Who uses the IF to pick a journal
(rate a candidate, etc.)?• Who knows how the IF is calculated
and from what data?
![Page 11: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/11.jpg)
The Impact Factor
A1 A2
C12
time
citationspublished
articlespublished
year 1 year 2 year 3
𝐼𝐹 (𝑦𝑒𝑎𝑟 3)=C12
A1+A2
Introduced in 1950’s by Eugene Garfield: ISI
![Page 12: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/12.jpg)
The Impact Factor
40 60
100
time
citationspublished
articlespublished
year 1 year 2 year 3
𝐼𝐹 (𝑦𝑒𝑎𝑟 3)=100
4 0+60=1
Introduced in 1950’s by Eugene Garfield: ISI
![Page 13: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/13.jpg)
The Impact Factor
Journal X IF 2010=
All citations from TR indexed journals in 2012 to papers in journal X
Number of citable articles published in journal X in 20010/11
€30,000-130,000/year subscription ratesCovers ~11,500 journals (Scopus covers ~16,500)
![Page 14: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/14.jpg)
Main Problems with the IF
• Negotiable
• Irreproducible
• Mathematically
unsound
![Page 15: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/15.jpg)
Negotiable
• PLoS Medicine, IF 2-11 (8.4)(The PLoS Medicine Editors (2006) The Impact Factor Game. PLoS Med 3(6): e291. http://www.plosmedicine.org/article/info:doi/10.1371%2Fjournal.pmed.0030291)
• Current Biology IF from 7 to 11 in 2003– Bought by Cell Press (Elsevier) in 2001…
![Page 16: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/16.jpg)
![Page 17: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/17.jpg)
![Page 18: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/18.jpg)
![Page 19: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/19.jpg)
![Page 20: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/20.jpg)
![Page 21: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/21.jpg)
Not Reproducible
• Rockefeller University Press bought their data from Thomson Reuters
• Up to 19% deviation from published records• Second dataset still not correct
Rossner M, van Epps H, Hill E (2007): Show me the data. The Journal of Cell Biology, Vol. 179, No. 6, 1091-1092 http://jcb.rupress.org/cgi/content/full/179/6/1091
![Page 22: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/22.jpg)
Not Mathematically Sound
• Left-skewed distributions• Weak correlation of individual article citation
rate with journal IF
Seglen PO (1997): Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314(7079):497 (15 February)http://www.bmj.com/cgi/content/full/314/7079/497
![Page 23: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/23.jpg)
Journal rank and citations
The weakening relationship between the Impact Factor and papers' citations in the digital age (2012): George A. Lozano, Vincent Lariviere, Yves Gingras arXiv:1205.4328
![Page 24: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/24.jpg)
Journal Rank and Quality
Brown, E. N., & Ramaswamy, S. (2007). Quality of protein crystal structures. Acta Crystallographica Section D Biological Crystallography, 63(9), 941–950. doi:10.1107/S0907444907033847
![Page 25: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/25.jpg)
Journal Rank and ‘Quality’
Munafò, M., Stothart, G., & Flint, J. (2009). Bias in genetic association studies and impact factor Molecular Psychiatry, 14 (2), 119-120 DOI: 10.1038/mp.2008.77
![Page 26: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/26.jpg)
Journal Rank and Methodology
Brembs, B., Button, K., & Munafò, M. (2013). Deep impact: unintended consequences of journal rank. Frontiers in Human Neuroscience, 7. doi:10.3389/fnhum.2013.00291
![Page 27: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/27.jpg)
NO EVIDENCE
Journal rank is a figment of our imagination.
![Page 28: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/28.jpg)
Journal Rank and Fraud/Error
Fang et al. (2012): Misconduct accounts for the majority of retracted scientific publications. PNAS 109 no. 42 17028-17033
![Page 29: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/29.jpg)
Journal Rank and Retractions
Data from: Fang, F., & Casadevall, A. (2011). RETRACTED SCIENCE AND THE RETRACTION INDEX Infection and Immunity DOI: 10.1128/IAI.05661-11
![Page 30: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/30.jpg)
Journal rank and citations
The weakening relationship between the Impact Factor and papers' citations in the digital age (2012): George A. Lozano, Vincent Lariviere, Yves Gingras arXiv:1205.4328
![Page 31: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/31.jpg)
INCENTIVES
“High-Impact” journals attract the most unreliable research
![Page 32: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/32.jpg)
![Page 33: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/33.jpg)
“Do you trust scientists?”
![Page 34: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/34.jpg)
“Who can you trust these days?”
![Page 35: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/35.jpg)
“Politicians? Financial experts? Realtors?“
![Page 36: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/36.jpg)
WHAT HAPPENED?
The disaster of our digital infrastructure
![Page 37: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/37.jpg)
HISTORY
Journal rank is a relic of the print era
![Page 38: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/38.jpg)
Distribution yesterday
![Page 39: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/39.jpg)
Subscriptions yesterday
![Page 40: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/40.jpg)
Meanwhile…
Modified from ARL: http://www.arl.org/bm~doc/arlstats06.pdf, http://www.arl.org/bm~doc/arlstat08.pdf
% C
han
ge
19861987
19881989
19901991
19921993
19941995
19961997
19981999
20002001
20022003
20042005
20062007
2008-50
0
50
100
150
200
250
300
350
400
Subscription pricesCPI/inflationJournals purchased
![Page 41: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/41.jpg)
Subscriptions today
![Page 42: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/42.jpg)
Distribution today
![Page 43: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/43.jpg)
SCHOLARSHIP
Institutions produce publications, data and software
![Page 44: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/44.jpg)
DISASTER I
Dysfunctional scholarly literature
![Page 45: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/45.jpg)
Literature
• No scientific impact analysis
• Limited access• No global search• No functional hyperlinks• No flexible data
visualization• No submission
standards• (Almost) no statistics• No text/data-mining• No effective way to sort,
filter and discover• No networking feature• etc.
…it’s like the web in 1995!
![Page 46: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/46.jpg)
DISASTER II
Scientific data in peril
![Page 47: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/47.jpg)
![Page 48: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/48.jpg)
![Page 49: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/49.jpg)
DISASTER III
Non-existent software archives
![Page 50: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/50.jpg)
![Page 51: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/51.jpg)
![Page 52: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/52.jpg)
Today‘s Digital Dystopia
• Institutional email• Institutional
webspace• Institutional blog• Library access card• Open access
repository
• No archiving of publications
• No archiving of software
• No archiving of data
![Page 53: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/53.jpg)
WHAT NOW?
Science, tear down this paywall!
![Page 54: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/54.jpg)
![Page 55: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/55.jpg)
1. International Coordination
![Page 56: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/56.jpg)
2. Hire software developers
![Page 57: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/57.jpg)
Superior Access
• Harvest all Open Access Publications– Accessible via single interface– Not just from green repositories– Everything not obviously illegal
• Integrate resulting database– PubMed– Google Scholar
• Plug the gaps:
![Page 58: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/58.jpg)
3. Cancel Subscriptions
![Page 59: The pernicious habit of ranking scientists by the journals they publish in](https://reader034.vdocument.in/reader034/viewer/2022042814/5549efe4b4c9050d488b52ad/html5/thumbnails/59.jpg)
Superior Alternative
• Global search and access for all literature, software and data
• Intelligent sort, filter and discover functionalities
• Scientific, evidence-based reputation system• Authoring tool for collaborative writing ans
single-click submission• Orders of magnitude cheaper: US$90/paper
(e.g. SciELO) vs. US$4,800/paper (subscription)