Overview & Methodology
The University for Business and the Professions
Peer Reviewing in Evolutionary
Computing
Dr. Peter W.H. SmithDepartment of Computing
The City UniversityNorthampton Square
LondonEC1V 0HB, England
Overview & Methodology
The University for Business and the Professions
Overview• Why do papers get rejected?
• Is Reviewing Biased? - If so how?
• Experience from Other Disciplines.
• Masked, open or double blind?
• The Cochrane Protocol
• Conclusions
Overview & Methodology
The University for Business and the Professions
Why Do Papers Get Rejected?• Because they’re crap!
• They are of no relevance to the conference/journal– the submission is outside the remit of the conference.– Theoretical paper submitted to an applications
conference and vice versa.– Paper flying under a flag of convenience.
• Significance of the problem– The work/results have been published elsewhere.– The work is of no conceivable interest or relevance.– The work adds nothing to our understanding of the
subject.
Overview & Methodology
The University for Business and the Professions
Why do papers get rejected?• Problems with Methodology.
– Poor scientific method.– Insufficient detail to reproduce results.– Poorly written up.– Scientifically flawed.
• The Idea is too New/Revolutionary.– Different ideas to mainstream/established orthodoxy.
(difficult to assess impact of ideas).
Overview & Methodology
The University for Business and the Professions
The Wrong Reasons for Rejecting Papers
• The author(s) is of the wrong gender/race/nationality.
• The Research demonstrates a negative result (e.g.
method X is superior to EC method Y).
• The research challenges an established orthodoxy.
• The research challenges established results (and
possibly the methodology of those who produced them.)
Overview & Methodology
The University for Business and the Professions
Bias in Peer Reviewing• Gender Bias - (Gilbert et al. 1994) found no evidence of this in the
publication of journal articles, although (Wenneras and Wold 1997)
showed that there was a peer review bias against women in grant
applications (in medicine).
• National Bias - (Link 1998) suggested a bias in favour of articles
written in the USA - but independent quality assessment of articles
submitted are required.
• Cronyism Bias - (Wenneras and Wold op. Cit.) also suggested that in
grant applications there was a bias in favour of applicants known to
panel members.
Overview & Methodology
The University for Business and the Professions
Bias in Peer Reviewing• Favouritism towards positive Results - the most important
challenge facing reviewers according to (Dickersin 1990)
and (Rennie 1998).
• Institutional Bias - (Harcum 1993) found that there was a
peer reviewing bias against researchers from small
departments in low prestige universities.
Overview & Methodology
The University for Business and the Professions
Masking of Author/Reviewer
Is Peer reviewing less biased when the author’s identity is withheld from reviewers? (Masking)What about the converse - the reviewer’s identity is not revealed to the author?
The Case For Masking: (Davidoff 1998)• Blinding is a central tenet of the methodology of clinical and scientific trials.• Bias is the demon of science - the opportunity for bias is reduced when experimental observations are not confounded by personal but irrelevant associations.
Overview & Methodology
The University for Business and the Professions
Masking and Blinding• Blinding has proven important in non-scientific judgements, e.g. selection of candidates for a symphony orchestra.•Bias can be made important to decision making, e.g. the use of celebrities in advertising (or big names on research papers :-)
•The Case against:• Not found necessary in journal reviewing (historical inertia).
Overview & Methodology
The University for Business and the Professions
Masking and Blinding
• Logistic problems - time consuming, expensive and technically difficult to achieve.•It is still an open question whether in practice it actually makes a difference. (McNutt et al. 1990) and (Fisher et al. 1994) found that masking an author’s identity improved reviewing in medicine. (Laband and Piette 1994) made similar findings in Economics. On the other hand (Van Rooyen et al. 1997) and (Justice et al. 1997) found no difference between masked and open reviews - there is a strong willingness to accept the null hypothesis.
Overview & Methodology
The University for Business and the Professions
Blinding The Reviewer’s Identity
The Case For:• Open reviewing inhibits reviewers from probing critiques - not in the interest of good science.
The Case Against:• Hiding behind anonymity fosters irresponsibility - slanted and destructive reviews (although Godlee et al. 1997) found that more problems were caused by a lack of critical faculty in reviewers).• Anonymous reviewers can become:
• inappropriately enthusiastic (zealots).• Overly harsh (assassins) (Siegelman 1991) - assassins also tend to nit-pick - grammar etc.
Overview & Methodology
The University for Business and the Professions
Masking/Blinding - Summary• A major aim of peer reviewing is objectivity - subjectivity
should be removed from this process as much as
possible.
• Blinded reviewers who act inappropriately can easily be
called to task - the editors know their identity.
• One medical journal uses a reviewing ombudsman (The
Lancet) - there is a strong case for an appeals process.
Overview & Methodology
The University for Business and the Professions
What is a Cochrane ReviewCochrane Reviews follow the following format:• Structured format - to help readers find their way around.• A detailed methods section - to assess whether the work was done in such a way to justify its conclusions.• Assessment of clinical methods by predefined criteria.• Thorough and systematic search strategy for references and other material.• Data summarised in a meta-analysis.• Easy to understand (for non-experts).• Reviews from multinational editorial teams, as well as “consumers”.
Overview & Methodology
The University for Business and the Professions
Cochrane Reviews - continued
•Use of glossaries.• Updateable reviews.• Reviews assessed according to their quality.•Comments and criticism invited to reviews.
Overview & Methodology
The University for Business and the Professions
Structure of a Cochrane Review• Synopsis - short summary.
• Structured Abstract.
• Background to research question.
• Objectives of the research.
• Selection criteria.– Types of studies.– Types of participants (who is interested in the results).– Types of interventions (comparison treatments).– Types of outcomes measures.
Overview & Methodology
The University for Business and the Professions
Cochrane Reviews• Search Strategy for identification of studies.
• Methods of the review.
• Description of studies.
• Methodological quality of included studies.
• Results - what does the data show - a meta-analysis.
• Discussion - interpretation and assessment of results.
• Reviewers’ conclusions
Overview & Methodology
The University for Business and the Professions
Conclusions and Suggestions• Make Reviewers aware of bias in the reviewing process.
• Allow a process of appeal against reviews.
• Structure submissions along the lines suggested by the Cochrane
protocol(?)
• Pre-register research ideas (thus defusing problems of negative
results).
• Encourage wider reviewing - who?
• Review reviewers.
• Encourage good practice among reviewers - structure the review to
discourage assassinations.
•