scholarly communications: changes to peer review bradley hemminger school of information and library...
Post on 20-Dec-2015
215 views
TRANSCRIPT
![Page 1: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/1.jpg)
Scholarly Communications:Changes to Peer Review
• Bradley Hemminger
• School of Information and Library Science
• University of North Carolina at Chapel Hill
![Page 2: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/2.jpg)
Scholarly Communications Process
Idea
V1
Present to colleagues
V2
Present at conference
V3
Submit to journal V4
Referees Revision for journal V5
Journal Final Revision
V6
Revision to update analysis
V7
Revision to include additional new results V8
![Page 3: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/3.jpg)
Scholarly Communications Process
Idea
V1
Present to colleagues
V2
Present at conference
V3
Submit to journal V4
Referees Revision for journal V5
Journal Final Revision
V6
Revision to correct analysis
V7
Revision to include additional new results V8
formulate discussion discussion,revision
Two peer reviews
CopyproofingCriticisms, new thoughts,revision
new results,revision
commentscommentscomments
Author revision
![Page 4: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/4.jpg)
Scholarly Communications Process: What’s Produced
Journal Final Revision
V6
![Page 5: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/5.jpg)
Scholarly Communications Process:What I’d like to see saved!Idea
V1
Present to colleagues
V2
Present at conference
V3
Submit to journal V4
Referees Revision for journal V5
Journal Final Revision
V6
Revision to correct analysis
V7
Revision to include additional new results V8
formulate discussion discussion,revision
Two peer reviews
CopyproofingCriticisms, new thoughts,revision
new results,revision
commentscommentscomments
Author revision
![Page 6: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/6.jpg)
Peer Review Output
Review
(Peer)
With Respect to XYZ…Accept reject revise
Comments to Author
Qualitative Grade
Qualitative Comments
Article
![Page 7: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/7.jpg)
Generalized Review Model
Review (open, peer, machine)
Accept,Reject,Revise,With respect to XYZ
Comments to Author
Qualitative Grade
Quantitative Grades
Score (1-10)
Qualitative Comments
Article
![Page 8: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/8.jpg)
Overview of Peer Review
Review
Peer, Open, Machine
Accept, reject, revise with respect to XYZ standards
Comments to Author
Qualitative Comments
Quantitative Grade
Published Article
Article submitted
Send elsewhere
Filter
Reject
Score (1-10)
Qualitative Grade
![Page 9: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/9.jpg)
General Review Model Parallels
• In general, you have sample (material) which is judged/scored quantitatively and qualitatively by an identified observer.
• Current Peer Review
• Automated Scoring Systems (lab tests)
• Moderated email lists (announce)
• Moderated Eprints servers (arXiv)
![Page 10: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/10.jpg)
Peer Review Options• Human Judgement
– Expert peer review (status quo)– Certified expert peer review– Open Peer Review BMJ, BioMed– Open comment review pyscprints
• Computer Judgement– Computer peer review
• Human Usage– Citation-based (CiteSeer)– Usage counts (CiteSeer) Example– Quantity of discussion
• Coarse Categorization– Two Tier (grey/gold)– Moderator (current arXiv)– No review (old arXiv)
Quantitative
Score (1-10)Score (1-10)Score (1-10)Score (1-10)
Score (1-10)
#citations#hits#number of
related discussions
QualitativeRel Yes/No GroupRel Yes/No GroupAbsolute Absolute
Rel Yes/No GroupRel Yes/No Group
YYYYY
?
![Page 11: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/11.jpg)
Judgment based on some combination of reviews/commentsIdea
V1
Present to colleagues
V2
Present at conference
V3
Submit to journal V4
Referees Revision for journal V5
Journal Final Revision
V6
Revision to correct analysis
V7
Revision to include additional new results V8
formulate discussion discussion,revision
Two peer reviews
CopyproofingCriticisms, new thoughts,revision
new results,revision
commentscommentscomments
Author revision
![Page 12: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/12.jpg)
What areas of improvement?
• Review Process Change
• Search, Retrieval Process Change
• Service Provider Process Change
![Page 13: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/13.jpg)
Review Process Changes
• Include open reviews and comments to get additional feedback.
• Support normal scholarly discourse, allowing give and take with author responding.
• Add quantitative scores to allow better filtering based on quality during retrieval
• Add machine (automated) reviews
![Page 14: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/14.jpg)
Search and Retrieval Changes
• Universal Archive: all material freely available.• Universal Searching: standardized metadata (
Dublin Core) for general searching.• Automated agents to bring material of interest to
your attention.• Use additional review scores (public reviews,
machine) to help filter search.• Example: article scores > 7.0, refereed,
citation count above X, type=research article, search terms = schizophrenia, geneX)
![Page 15: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/15.jpg)
Provider Service Change
• What is worth paying for?– Quality review (Faculty of 1000)– Proofing, citation linking, professional presentation (
CiteSeer, Cite-base)– Archival (JStor)
• Who hosts material:– Society (arXiv)– Commerical Publishers (Elesiever,BioMedCentral)– University Library (MIT Dspace)
![Page 16: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/16.jpg)
New Frameworks for Peer Review
• As an enabling technology: frameworks like NeoRef supports all of the above models in any combination at the same time, while eliminating many of the costs.
• Requirements: Based on OAI and Dublin Core, and expectation of logical universal archive, an universal unique object IDs (URL, DOIs) and person IDs.
![Page 17: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/17.jpg)
Example Model (NeoRef)• All material and metadata are author contributed to a
public OAI archive (author retains ownership).• All materials universally available via search engines that
harvest metadata from OAI archives.• OAI archives have automated or manual moderator to
filter out “junk”.• Everything--articles, reviews, comments, indexings, etc.,
are stored as digital content on archive using the same mechanism. Reviews contain quantitative score, qualitative grade, qualitative comments. Logically (although not physically), a two tier (Grey & Gold) system for materials– High quality keep forever material reviewed by known entity– Grey material (everything else)
![Page 18: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/18.jpg)
NeoRef for Movies, Dates, DocSouth
• The same process used by NeoRef to support Scholarly Communication could be used for almost any purpose. All that is required is storage of Digital Content Items, and linking of reviews, comments, etc to them.
• Movies: Grey is everyone’s reviews; Gold is Siskel and Ebert reviews
• DocSouth: self cataloged and indexed items are Grey; librarian/archivist cataloged and indexed items are Gold.
![Page 19: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/19.jpg)
Can we save the Gold and Grey?
Idea
V1
Present to colleagues
V2
Present at conference
V3
Submit to journal V4
Referees Revision for journal V5
Journal Final Revision
V6
Revision to correct analysis
V7
Revision to include additional new results V8
formulate discussion discussion,revision
Two peer reviews
Author revision
Criticisms, new thoughts,revision
new results,revision
commentscommentscomments
Copyproofing
![Page 20: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/20.jpg)
NeoRef Storage Model
Conference paper (v3)
Comments on V6
Journal Submission V4
Journal Final Revision V6
Revision to include additional results and analyses V8
Auto-indexing
Material expressing content
Two peer reviewsLocal powerpoint Presentation v2
Comments on V3
Automated
Author Indexing
Recognized Expert
Open (anyone)Top Tier (Keep Forever)
Filter (Moderate)
Grey Literature
Author
Machine Review
![Page 21: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/21.jpg)
What do users want?
![Page 22: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/22.jpg)
The Association of Learned and Professional Society Publishers (ALPSP) Survey
Authors and Electronic Publishing• Scholarly research communication has seen far-
reaching developments in recent years. • Most journals are now available online as well as
in print, and numerous electronic-only journals have been launched;
• the Internet opens up new ways for journals to operate.
• Authors have also become conscious of alternative ways to communicate their findings, and much has been written about what they ought to think.
![Page 23: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/23.jpg)
ALPSP felt that it would be timely to discover what they actually thought and what they actually did. This survey aimed to discover the views of academics, both as authors and as readers. Some 14,000 scholars were contacted across all disciplines and all parts of the world, and nearly 9% responded; their detailed comments make thought-provoking reading.
Alma Swan and Sheridan Brown. Authors and Electronic Publishing: The ALPSP Research Study on Authors' and Readers’ Views of Electronic Research Communication. (West Sussex, UK: The Association of Learned
and Professional Society Publishers, 2002).http://www.alpsp.org/pub5.htm
![Page 24: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/24.jpg)
Importance of the Peer Review Process
0102030405060708090
100Peer-reviewed
Refs' commentspublished
Referees identified
Public commentary oneprints
Post-publication publiccommentary
Ability to submitcomments
http://www.alpsp.org/pub5.ppt
![Page 25: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/25.jpg)
Importance of journal features
010
20
30
40
50
60
70
80
90
Citation links Additionaldata
Addit/colourimages
Manipulablecontent
Video/sound
![Page 26: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/26.jpg)
Importance of the peer review process
0102030405060708090
100Peer-reviewed
Refs' commentspublished
Referees identified
Public commentary oneprints
Post-publication publiccommentary
Ability to submitcomments
![Page 27: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/27.jpg)
Importance of publishers’ rolesFactor Responses as authors Responses as readers
Peer review 81 80
Gathering articles together to enable browsing of content
64 49
Selection of relevant and quality-controlled content
71 54
Content editing and improvement of articles
60 39
Language or copy editing 50 34
Checking of citations/adding links
46 28
Marketing (maximising visibility of journal)
44 20
![Page 28: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/28.jpg)
Importance of future dissemination channels
Dissemination method Very important plus important
categories
Ranking
Traditional print + electronic journal 91 1
Discipline-based electronic reprint archive 78 2
Traditional print journal 77 3
Traditional electronic-only journal 66 4
Institution-based electronic reprint archive 60 5
New forms of electronic-only journal 49 6
Discipline-based electronic preprint archive
44 7
Institution-based electronic preprint archive
33 8
![Page 29: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/29.jpg)
http://www.update-software.com/Cochrane/MR000016.pdf
![Page 30: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/30.jpg)
Cochrane Methodology Review
• Despite its widespread use and costs, little hard evidence exists that peer review improves the quality of published biomedical research.
• There had never even been any consensus on its aims and that it would be more appropriate to refer to it as ‘competitive review’.
Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,”BMJ 326 (February 1, 2003): 241
http://bmj.com/cgi/reprint/326/7383/241/a.pdf
![Page 31: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/31.jpg)
Cochrane Methodology Review• On the basis of the current evidence, ‘the
practice of peer review is based on faith in its effects, rather than on facts,' state the authors, who call for large, government funded research programmes to test the effectiveness of the [classic peer review] system and investigate possible
alternatives. Caroline White, “Little Evidence for Effectiveness of Scientific Peer Review,”
BMJ 326 (February 1, 2003): 241http://bmj.com/cgi/reprint/326/7383/241/a.pdf
![Page 32: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/32.jpg)
Cochrane Methodology Review• The use of peer-review is usually
assumed to raise the quality of the end-product (i.e. the journal or scientific meeting) and to provide a mechanism for rational, fair and objective decision-making. However, these assumptions have rarely been tested.
Tom O. Jefferson, Phil Alderson, Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of
Biomedical Studies. (Middle Way, Oxford:Update Software Ltd, 2003).
http://www.update-software.com/Cochrane/MR000016.pdf
![Page 33: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/33.jpg)
Cochrane Methodology Review• The available research has not clearly
identified or assessed the impact of peer-review on the more important outcomes (importance, usefulness, relevance, and quality of published reports)
• … [G]iven the widespread use of peer-review and its importance, it is surprising that so little is known of its effects
Tom O. Jefferson, Phil Alderson,Frank Davidoff, and Elizabeth Wager, Editorial Peer-review for Improving the Quality of Reports of
Biomedical Studies. (Middle Way, Oxford:Update Software Ltd, 2003).
http://www.update-software.com/Cochrane/MR000016.pdf
![Page 34: Scholarly Communications: Changes to Peer Review Bradley Hemminger School of Information and Library Science University of North Carolina at Chapel Hill](https://reader030.vdocument.in/reader030/viewer/2022032800/56649d455503460f94a22384/html5/thumbnails/34.jpg)
FURTHERMORE …• 16% said that the referees would no
longer be anonymous• 27% said that traditional peer review
would be supplemented by post-publication commentary
• 45% expected to see some changes in the peer-review system within the next five years
Fytton Rowland, “The Peer-Review Process,” Learned Publishing 15 no. 4 (October 2002): 247-258.
Report version: http://www.jisc.ac.uk/uploaded_documents/rowland.pdf