the university for business and the professions overview & methodology peer reviewing in...

Overview & Methodology

The University for Business and the Professions

Peer Reviewing in Evolutionary

Computing

Dr. Peter W.H. SmithDepartment of Computing

The City UniversityNorthampton Square

LondonEC1V 0HB, England



Overview• Why do papers get rejected?

• Is Reviewing Biased? - If so how?

• Experience from Other Disciplines.

• Masked, open or double blind?

• The Cochrane Protocol

• Conclusions



Why Do Papers Get Rejected?• Because they’re crap!

• They are of no relevance to the conference/journal– the submission is outside the remit of the conference.– Theoretical paper submitted to an applications

conference and vice versa.– Paper flying under a flag of convenience.

• Significance of the problem– The work/results have been published elsewhere.– The work is of no conceivable interest or relevance.– The work adds nothing to our understanding of the

subject.



Why do papers get rejected?• Problems with Methodology.

– Poor scientific method.– Insufficient detail to reproduce results.– Poorly written up.– Scientifically flawed.

• The Idea is too New/Revolutionary.– Different ideas to mainstream/established orthodoxy.

(difficult to assess impact of ideas).



The Wrong Reasons for Rejecting Papers

• The author(s) is of the wrong gender/race/nationality.

• The Research demonstrates a negative result (e.g.

method X is superior to EC method Y).

• The research challenges an established orthodoxy.

• The research challenges established results (and

possibly the methodology of those who produced them.)



Bias in Peer Reviewing• Gender Bias - (Gilbert et al. 1994) found no evidence of this in the

publication of journal articles, although (Wenneras and Wold 1997)

showed that there was a peer review bias against women in grant

applications (in medicine).

• National Bias - (Link 1998) suggested a bias in favour of articles

written in the USA - but independent quality assessment of articles

submitted are required.

• Cronyism Bias - (Wenneras and Wold op. Cit.) also suggested that in

grant applications there was a bias in favour of applicants known to

panel members.



Bias in Peer Reviewing• Favouritism towards positive Results - the most important

challenge facing reviewers according to (Dickersin 1990)

and (Rennie 1998).

• Institutional Bias - (Harcum 1993) found that there was a

peer reviewing bias against researchers from small

departments in low prestige universities.



Masking of Author/Reviewer

Is Peer reviewing less biased when the author’s identity is withheld from reviewers? (Masking)What about the converse - the reviewer’s identity is not revealed to the author?

The Case For Masking: (Davidoff 1998)• Blinding is a central tenet of the methodology of clinical and scientific trials.• Bias is the demon of science - the opportunity for bias is reduced when experimental observations are not confounded by personal but irrelevant associations.



Masking and Blinding• Blinding has proven important in non-scientific judgements, e.g. selection of candidates for a symphony orchestra.•Bias can be made important to decision making, e.g. the use of celebrities in advertising (or big names on research papers :-)

•The Case against:• Not found necessary in journal reviewing (historical inertia).



Masking and Blinding

• Logistic problems - time consuming, expensive and technically difficult to achieve.•It is still an open question whether in practice it actually makes a difference. (McNutt et al. 1990) and (Fisher et al. 1994) found that masking an author’s identity improved reviewing in medicine. (Laband and Piette 1994) made similar findings in Economics. On the other hand (Van Rooyen et al. 1997) and (Justice et al. 1997) found no difference between masked and open reviews - there is a strong willingness to accept the null hypothesis.



Blinding The Reviewer’s Identity

The Case For:• Open reviewing inhibits reviewers from probing critiques - not in the interest of good science.

The Case Against:• Hiding behind anonymity fosters irresponsibility - slanted and destructive reviews (although Godlee et al. 1997) found that more problems were caused by a lack of critical faculty in reviewers).• Anonymous reviewers can become:

• inappropriately enthusiastic (zealots).• Overly harsh (assassins) (Siegelman 1991) - assassins also tend to nit-pick - grammar etc.



Masking/Blinding - Summary• A major aim of peer reviewing is objectivity - subjectivity

should be removed from this process as much as

possible.

• Blinded reviewers who act inappropriately can easily be

called to task - the editors know their identity.

• One medical journal uses a reviewing ombudsman (The

Lancet) - there is a strong case for an appeals process.



What is a Cochrane ReviewCochrane Reviews follow the following format:• Structured format - to help readers find their way around.• A detailed methods section - to assess whether the work was done in such a way to justify its conclusions.• Assessment of clinical methods by predefined criteria.• Thorough and systematic search strategy for references and other material.• Data summarised in a meta-analysis.• Easy to understand (for non-experts).• Reviews from multinational editorial teams, as well as “consumers”.



Cochrane Reviews - continued

•Use of glossaries.• Updateable reviews.• Reviews assessed according to their quality.•Comments and criticism invited to reviews.



Structure of a Cochrane Review• Synopsis - short summary.

• Structured Abstract.

• Background to research question.

• Objectives of the research.

• Selection criteria.– Types of studies.– Types of participants (who is interested in the results).– Types of interventions (comparison treatments).– Types of outcomes measures.



Cochrane Reviews• Search Strategy for identification of studies.

• Methods of the review.

• Description of studies.

• Methodological quality of included studies.

• Results - what does the data show - a meta-analysis.

• Discussion - interpretation and assessment of results.

• Reviewers’ conclusions



Conclusions and Suggestions• Make Reviewers aware of bias in the reviewing process.

• Allow a process of appeal against reviews.

• Structure submissions along the lines suggested by the Cochrane

protocol(?)

• Pre-register research ideas (thus defusing problems of negative

results).

• Encourage wider reviewing - who?

• Review reviewers.

• Encourage good practice among reviewers - structure the review to

discourage assassinations.

•

the university for business and the professions overview & methodology peer reviewing in...

Documents