1

Thoughts on how to improve reviews

Paul Francis

(Presented by Robbert Van Renesse)

Cornell

2

Main problem with current system Reviews are often of poor quality

I suspect main reason is that PC members are overworked

Both ideas in this presentation are focused on this one problem alone

3

Idea One:

Allow PC members to ask authors simple “where in the paper can I find this?” questions

Authors can answer ONLY with page, column, and line numbers

Why? Often when I reject a paper, it is based on one or

a few specific flaws. It would be good to verify that I’m not overlooking something.

4

A reviewer comment

“But you can do this today...just ask the PC chairs to forward an email”

My rebuttal:

1. Bothering the PC chairs is a significant deterrent

2. Should really limit author reply to text already in paper...otherwise author is effectively submitting more than what is in the paper

5

Today

6

Proposal One: Pass reviews on

7

Observations

Reviewer gets the benefit of previous reviewers

Various ways this can be exploited: To improve review: Reviewer first does complete

unbiased review, then uses previous reviews as sanity check

To reduce work: Reviewer scans paper, previous reviews, and rebuttal, and then does only enough work to decide if author has overcome previous criticisms This can be done in good faith

8

Observations

Good paper should be able to overcome earlier criticisms Without having to torque paper to satisfy future

reviewers.... Instead author can use rebuttal

Truly bad paper should not be able to overcome earlier criticisms Prevents bad paper from retrying until it gets lucky Might discourage “hail-mary” submissions

9

A reviewer comment

“This will promote reviewer laziness” My rebuttal:

A good-faith reviewer will benefit from earlier reviews

A lazy reviewer already has many ways of being lazy (form instant opinion, do minimal work to support opinion...)