9/19/2011
1
PEER REVIEW
Bob GarryIllustration:James Wang Paradigm Magazine
9/19/2011
2
license agreement: cartoonstock
9/19/2011
3
9/19/2011
4
NIH Study Section Service•Member of AIDS Study Section (SS) 1988-90•Member of AIDS Mol. Biol. SS (1990-96)•Chair, VATID Biodefense SS (2002)•Chair, SBIR-STTR Biodefense SS (2002-06)•Co-Chair Regional Centers of Excellence for Biodefense SSs (2008)•Chair: New Approaches to HIV Vaccines SS (2009, 2010)•Member “Bugs and Drugs” SS (2007 – date)
9/19/2011
5
Definitions• Peer
[1] Someone “of equal standing with another … especially belonging to the same societal group…or [having the same] status” [2] “A noble with a hereditary title, i.e., a peerage, and in times past, with certain rights and privileges not enjoyed by commoners.”
• Review[1] A critical inspection or examination[2] A second or repeated viewing of past events, circumstances or facts.
Wicktionary
9/19/2011
6
Peer Review is performed by:• Publishers: Manuscripts (publish or
perish)• Agencies or foundations: Grants (follow
the money)• The Feds (FDA, CDC, USDA, EPA,
patent office, etc.)• Institutional committees (IRBs, IACUC,
IBC, promotions and tenure, misconduct, etc.)
9/19/2011
7
EDTIOR
REVIEWERS
INPUT
PEER REVIEW
PEERREVIEW
OUTPUT
Submission of manuscripts, grant applications, etc.
Publications, funded grants, etc. OR heartache and rejection
9/19/2011
8
Traditional publishing houses
Elsevier Wiley Springer
40% of all journals are published by the “big three.”
Nature
9/19/2011
9
“Society” journals
9/19/2011
10
Federal Research Public Access Act
Each agency of >$100 million must:• Require researchers to submit an
electronic copy of published papers.• Ensure that the manuscript is
preserved in a stable digital repository.• Require that free, online access for
manuscript occurs within than six months.
9/19/2011
11
The Federal Research Public Access Act does not apply to:
• laboratory notes, preliminary data analyses, author notes, phone logs, or other information used to produce the final manuscript.
• classified research or research that results in works that generate revenue or royalties for the author (such as books). Patentable discoveries are exempt only to the extent necessary to protect copyright or a patent.
9/19/2011
12
Open Access Publishers
9/19/2011
13
Open access to articles: 2009Björk B-C, Welling P, Laakso M, Majlender P, Hedlund T, et al. (2010) Open Access to the Scientific Journal Literature: Situation 2009. PLoS ONE 5(6): e11273. doi:10.1371/journal.pone.0011273
9/19/2011
14
Peer Review: limitations/problems
• slow• expensive• time consuming • highly subjective• prone to bias• easily abused• poor at detecting gross defects• almost useless for detecting fraud.
9/19/2011
15
Nobel winning science:Problems or outright rejection
Rejecting and resisting Nobel class discoveries:accounts by Nobel LaureatesJUAN MIGUEL CAMPANARIOScientometrics, Vol. 81, No. 2 (2009) 549–565
9/19/2011
16
Nature rejection of Kreb’s cycle manuscript
Hans A. Krebs, PhD(1900-1981)Nobel Prize, 1953
Rejecting and resisting Nobel class discoveries:accounts by Nobel LaureatesJUAN MIGUEL CAMPANARIOScientometrics, Vol. 81, No. 2 (2009) 549–565
9/19/2011
17
Sir Winston ChurchillBritish politician (1874 - 1965)
“It has been said that democracy is the worst form of government except all the others that have been tried.”
“It has been said that peer review is the worst system of scientific accountability except all the others that have been tried.”
9/19/2011
18
9/19/2011
19
Open review(“Where everybody knows your name”)
• Reviewers sign their reviews.
• Has also been used to describe a process whereby community scientists can post (signed or unsigned) reviews –better name: Public review.
9/19/2011
20
Open reviewing: Pro• Maintains balance ( judgemental role vs.
helping authors• Adds credibility• Reviewers become more accountable• Eliminate abuses of the system (maybe)• Little justification for secrecy• Less disagreeable/more polite (maybe)
9/19/2011
21
Open reviewing: Cons• Reprisal, especially by big “Kahunas”• Creates an “old boy” network • Resentment and animosity• Higher acceptance rate?• Not broke - don’t fix it
9/19/2011
22
The Internet permits more interactive science publishing
• Peer review need no longer be a one-step process
• Online comment systems allow analysis after an article is published.
• Some journals use a two-stage system formally (for ex. Physics).
9/19/2011
23
Nature's public review trial –aka open comment (2006)
• 1,369 papers were received for review during the trial period.
• 71 authors (or 5%) agreed to their papers being displayed for open comment.
• 33 received no comments; 38 (54%) received 92 technical comments (more than half [49] were to 8 papers).
9/19/2011
24
The future of scientific publishing• Several models are being investigated:
eg. open publication and archiving.• Many journals are adding community-
oriented features. • Online scientific interaction outside
journals (Science Blogs) is becoming increasingly important
Hendler, J./ Reinventing Academic Publishing-Part 1;IEEE Intelligent Systems 22, 2-3, 2007.
9/19/2011
25
Should I review the paper?
Am I qualified?
Do I have a Conflict of Interest?
Do I have the Time?
9/19/2011
26
Peer Review of Manuscripts Do’s
• Support work of high quality• Set the standards of the journal and of the
field • Consider the appropriateness of the paper
for the journal • Appropriately challenge flawed papers (but
be collegial). • Provide your scientific expertise (editorial
assistance – rewriting paper not needed)
9/19/2011
27
Peer Review of Manuscripts Don’ts
• Don’t give your scientific stamp of approval to a flawed paper.
• Don’t demean the authors.• Don’t agree to review unless you have the time
(and don’t delay papers on purpose or not) • Don’t pass the paper on to someone else to
review? • Don’t contact the author about the work or the
paper? • Don’t undermine the confidentiality of the review
process.
9/19/2011
28
Peer review at NIH: Process Overview
• The goal: evaluate applications in a way that is fair, equitable, timely, and free of bias.
• Review is a TWO-step process:1.Scientific Review Group (SRG). 2.Institute and Center (IC) National Advisory
Councils or Boards. • Only applications favorably recommended
by both the SRG and the Advisory Council may be recommended for funding.
9/19/2011
29
NIH CONFLICT OF INTEREST RULES:
• Employment• Financial Benefit • Personal Relationships (Relatives)• Professional Associates • Standing Review Group Membership • Longstanding Disagreements • Multi-Site Or Multi-Component Project • Request For Applications (RFA) Or Request
For Proposals (RFP)
9/19/2011
30
NIH CONFIDENTIALITY AND NON DISCLOSURE
RULES: • You must sign a CONFIDENTIALITY
AND NON-DISCLOSURE OF MATERIALS AND PROCEEDINGS agreement to participate in an NIH review.
9/19/2011
31
9/19/2011
32
Study Section: Seating chartProgram Staff
Scientific Review Officer (SRO)
Chair
Administrative AssistantReviewers
Telephone reviewers
http://www.youtube.com/watch?v=HMO3HoLJuJY
9/19/2011
33
(Fr. trier, separate, sort, sift or select) all applications are reviewed, but generally only a fraction (40-60%) are discussed
Triage
9/19/2011
34
Scored Review Criteria Significance
• Is an important problem or a critical barrier addressed?
• Will knowledge, technical capability, and/or clinical practice be improved?
• Will the proposed research change concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?
9/19/2011
35
Scored Review Criteria Investigator(s)
• Are researchers well suited to the project? • Do Early Stage Investigators or New
Investigators have appropriate experience and training? Have established investigators demonstrated an ongoing record of accomplishments?
• If collaborative or multi-PD/PI, do the investigators have complementary/integrated expertise; is leadership approach appropriate?
9/19/2011
36
9/19/2011
37
Scored Review Criteria Innovation
• Will current research or clinical practice paradigms be shifted?
• Are the concepts, approaches or methodologies, instrumentation, or interventions novel?
• Is a refinement, improvement, or new application of theoretical concepts, approaches or methodologies, instrumentation, or interventions proposed?
9/19/2011
38
Schachman H K J. Biol. Chem. 2006;281:6889-6903
9/19/2011
39
Scored Review Criteria - Approach
• Will approaches accomplish the specific aims of the project?
• Are problems, alternative strategies, and benchmarks for success presented?
• Will the strategy establish feasibility and will particularly risky aspects be managed?
• Are human subjects minorities children, etc. justified
9/19/2011
40
Scored Review Criteria Environment
• Does the scientific environment contribute to the probability of success?
• Is institutional support, equipment and other physical resources adequate?
• Will the project benefit from unique features of the scientific environment, subject populations, or collaborative arrangements?
9/19/2011
41
9/19/2011
42
Additional Review Criteria(Scored)
• Protections for Human Subjects• Inclusion of Women, Minorities, and
Children• Vertebrate Animals• Biohazards• Resubmission• Renewal• Revision
9/19/2011
43
Additional Review Considerations (unscored)
• Budget• Applications from Foreign Organizations• Select Agents• Resource Sharing Plans• Period of Support
9/19/2011
44
Overall Impact• “Reviewers will provide an overall
impact/priority score to reflect their assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s) involved, in consideration of the scored review criteria, and additional review criteria (as applicable for the project proposed).”
9/19/2011
45
New scoring (10-90)
9/19/2011
46
Second Level of Review • NIH program staff members examine
overall impact/priority scores, percentile rankings and summary statements and consider IC's needs.
• Program staff provide a grant-funding plan to Council.
• Council considers IC’s goals and needs and advises the IC director.
• IC director makes final funding decisions.
9/19/2011
47
9/19/2011
48
9/19/2011
49
Peer Review of Grants Do’s
• Support work of high quality• Appropriately challenge flawed
applications• Be collegial to the applicants and
the other members of the study section and program officers.
• Provide your scientific expertise.• Spend enough time writing and
discussing each application.
9/19/2011
50
Peer Review of Grants Don’ts
• Don’t give your scientific stamp of approval to a flawed application.
• Don’t demean the applicants, the other members of the SS or program officers.
• Don’t let your students write the review? • Don’t contact the applicants about the
work or the application? • Don’t undermine the confidentiality of the
review process.
9/19/2011
51
Sir Winston ChurchillBritish politician (1874 - 1965)
“It has been said that peer review is the worst system of scientific accountability except all the others that have been tried.”
W.C. FieldsAmerican comedian(1880 - 1946)
““If at first you don't succeed, try, try again. Then quit. There's no point in being a damn fool about it.”
9/19/2011
52
Ethical concerns may arise during peer review
9/19/2011
53
9/19/2011
54
http://ori.hhs.gov/education/products/niu_authorship/mistakes/index.htm#
9/19/2011
55
“On one hand, reviewers and editors must take all appropriate steps to preclude publication of duplicate, plagiarized or fraudulent papers. On the other hand, the suspicion of scientific misconduct can have a devastating impact on a scientific career, even if deliberate malevolence is eventually disproved. Because of this, the reviewer should carefully review the facts underlying his/her concerns.”
Sara Rockwell, Ph.D. Yale University School of Medicine
9/19/2011
56
– “Carefully reviewed studies sometimes turn out to be wrong because later attempts at repetition fail. But peer review requires authors to provide more data and more confirming material, making it likelier that careful efforts at confirmation will follow. Fraud is something quite different, and very hard to detect… The reporting of scientific results is based on trust. It's better to trust our colleagues, despite the fact that on rare occasions one of them might disappoint other scientists and those hoping for cures.”
Donald Kennedy , Editor-in-Chief of Science. Science 2006: Vol. 311; no. 5758, p. 145 -on the occasion of the Woo Suk Hwang scandal
9/19/2011
57
Research Misconduct NIH reviewers
“Research misconduct is defined as fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results, but not honest error or differences of opinion. It is vital that you do not make allegations of potential misconduct in the critique. Instead, such concerns must be brought to the attention of the SRO in a confidential manner, preferably before the study section meets.”
9/19/2011
58
9/19/2011
59
• Dr. XXX has entered into a Voluntary Exclusion Agreement in which she has voluntarily agreed, for a period of three (3) years, beginning on July 22, 2010:
1. To exclude herself from any contracting or subcontracting with any agency of the U.S. Government and from eligibility for, or involvement in, nonprocurement programs of the U.S. Government referred to as ``covered transactions'' pursuant to the HHS Implementation of OMB Guidelines to Agencies on Governmentwide Debarment and Suspension at 2 CFR 376, et seq.; and
2. To exclude herself from serving in any advisory capacity to PHS, including but not limited to service on any PHS advisory committee, board, and/or peer review committee, or as a consultant.
Posted on http://ori.hhs.gov/ and sent to federal grant recipients via email
9/19/2011
60
.
Schachman H K J. Biol. Chem. 2006;281:6889-6903©2006 by American Society for Biochemistry and Molecular Biology
PLEASE – DO NOT BE THIS GUY!!!