1

1

The peer review process and the task of a referee

Olli SilvénDepartment of Computer Science and Engineering

FI-90014 University of Oulu, Finland

2

Introduction (1)

A scientific paper is expected to provide a sufficient contribution to the knowledge base of its field

• Number of scientific papers and articles since 1665: >50 million (*) (the first ones in ’Philosophical Transactions of the Royal Society’)

• About 85% in the fields of science and technology ~ thousands of pages/day

The number of papers and articles submitted for publication is much larger• refereeing process selects the ones to be published

Examples of acceptance rates after refereeing:• IEEE journals: ~10-40% (large variance)• IEEE conferences: ~10-70% (very large variance)

(lowest acceptance rates for oral presentations <5%!)• IEEE workshops: ~30%-90%

Refereeing is also used in selecting research projects to be funded

(*) Arif Jinha (2010). Article 50 million: an estimate of the number of scholarly articles in existence, Learned Publishing, 23 (3)

2

3

Introduction (2)

What is a sufficient contribution?

• new result, theoretical or experimental e.g. ”we derive a method for estimating cross-correlations in complex lapped transform (CLT) domain”

• new insight

• novel synthesis of idease.g. magnetic resonance imaging

• useful survey• useful tutorial

4

Introduction (3)

What is not a sufficient contribution• new, novel, useful• badly written• erroneous data• non-sensical results

(for a famous published example see: Alan D. Sokal, Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity,in Social/Text #46/47, pp. 217-252 (Spring/Summer 1996))

MPI = Minimum Publishable Increment depends on the forum

3

5

The Sokal affair: an excerpt of the paper

6

Peer review process

Peer reviews are carried out by anonymous referees who evaluate the sufficiency of contribution

• novelty, significance, correctness, readability

Refereeing is public service to the scientific community• professional obligation, • carried out on volunteer basis• requires high expertice• helps in improving one’s own expertice• ensures the integrity of science

4

7

Peer review process of a journal

editor

associateeditors

author

referees

acceptrejectrevise

submission

reviewsrecommendations

selection of refereeschecking of revised papers

publish

selection of associate editor

8

Peer review process of a conference

program chairprogram committee author

referees

accept/reject/(conditional) accept with revisions

submission

accept/reject/minor revision recommendations

selection of therefereeschecking ofrevisions

extra referees

5

9

Peer review process of a conference with rebuttal process (1)

Rebuttal process is used in some conferences• imitates journal review process with the following steps

1. submission2. review 3. authors’ rebuttals to reviewers’ comments 4. review update by reviewers, or by assigned

Program Committee Members

The objective is to improve the quality of the review process• reveals unreliable reviews• encourages reviewers to react to authors’ rebuttals in their reviews

The downside:• heavy process, tight deadlines

10

Peer review process of a conference with rebuttal process (2)

program chairprogram committee

author

referees

accept/ reject/accept with revisions

submission

accept/reject/minor revision recommendations

selection of therefereeschecking ofrevisions

extra referees

review commentsrebuttal & revisions

6

11

Peer review process of a workshop

program chairprogram committee author

extra referees

accept/reject

submit

refereeingchecking ofrevisions

12

Peer review process of a research programme

steeringcommittee proposer

referees

accept with partial funding/reject

submission

Notice: not representative of TEKES or EU research programmes

7

13

The tasks of a referee

The reviewer grades a paper based on its novelty, significance,correctness, and readability

1. In case of substantial conflicts of interest or if the paper is out of the field of the reviewer, the editor must be informed promptly

2. Both positive and negative findings are summarized in a referee report

* confidential part only for the editor/program committee:information that could reveal the identity of the reviewer or in minor conflicts of interest

* non-confidential part for the author/editor/program committee

3. Learn from the other reviews, if they are sent to you after the process

14

The right attitude: I can learn something!

Humbleness and an open mind needed; 100% self-confidence can be harmful

Early assumptions on the correctness of the paper or the sufficiency of its references should be avoided

• an elegantly written paper may have zero actual contribution• a paper with broken English may contain a major new idea

The papers recommended for acceptance should have novelty and be correct• If the reviewer can’t check a fact or is unsure, this should be stated in the report

But don’t waste your time on analysing in detail a paper that is never publishable• a single crucial error is enough

8

15

Reviewing a research paper

The paper to be reviewed is typically accompanied with a review form• fill the five point scale questions last• it is most important to write an itemized review report

1. Relevance [ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent

2. Originality [ ] poor [ ] marginal [ x] fair [ ] good [ ] excellent

3. Background knowledge of the subject and references [ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent

4. Technical content [ ] poor [ ] marginal [ ] fair [ x ] good [ ] excellent

5. Presentation [ ] poor [ ] marginal [ x] fair [ ] good [ ] excellent

16

Reviewing a research paper: analysisThe analysis of a paper can be done by generating explanations to the following eight points (Smith 1990)

1. What is the purpose of the paperIs the problem clearly stated and have the key issues been pointed out?Is it clear what has been accomplished?

2. Is the paper appropriate for the intended forum?If it is not, what could be a better choice?

3. Is the goal significant = has the work been worth doing? Are the results just trivial variations or extensions of previous results?Are there any new ideas, or novelties in research methodology?What is the overlap with the previous papers by the same authors?

Citation analysis using electronic libraries are a big help!

9

17

Reviewing a research paper: the degree of self-plagiarism

Self-plagiarism = an author re-uses parts from his previous publications in later manuscripts• sometimes the manuscript is just a re-titled and reformatted version of an earlier paper• usually the manuscript is assembled from segments of earlier papers

The allowed degree of self-plagiarism depends on the forum• some IEEE journals do not allow a single sentence verbatim overlap without citation• IEEE conference papers can be used as a basis for a full journal paper submission;

- the authors are still required to cite related prior work• the papers cannot be identical and the journal paper must include sufficient novelties• in the paper the authors must clearly indicate the differencies to te previous work• with proper citations the maximum self-plagiarism is 25% - sanctions after that

The sanctions: immediate rejection of the current manuscriptwithdrawal of all manuscripts of all co-authors to any IEEE forumprohibition against each author for any new submissions to any IEEE forum

18

Example of analysing a research paper (1)

10

19

Background of the paper

Most video coding standards rely on hybrid block based transform coding methods. A key technique in achieving good compression efficiency is motion estimation. The best method is ”full-search” via correlation, but it is very expensive. Other techniques compromise compression efficiency, but reduce computational complexity.

20

Example (2)

Understand the purpose of the paper = read the abstract and introduction

11

21

Example (3)

Is the goal significant = has the work been worth doing?

O(N2*M2)->O(M2), N typically 16, M typically 48*48

22

Reviewing a research paper: analysis (cont’d)

4. Is the method of approach clear and valid?Is there something fundamentally flawed in the approach? Are the assumptions realistic and does that matter?Is the method new? Can it be generalized to other problems?

Again, electronic libraries are most useful.

5. Is the actual execution of the research correct?Are the mathematics and statistics correct? Check!Have the simulations been described in sufficient detail for replication?What about the boundary conditions?Do the results make sense?

This part may require considerable effort from the reviewer...

12

23

Example (4)

Is the method of approach clear and valid? Now, it is a mathematical one: Section II, Equation (3)

24

Reviewing a research paper: analysis (cont’d)

6. Are the conclusions correct?Are he results analysed to an adequate depth?What are the applications or implications of the results?

7. Is the presentation satisfactory?Is the paper readable? Is it structured according to the conventionsof scientific publications?

8. What did you as the reviewer learn?If you didn’t learn anything, then the paper is not publishable(provided that you understood the paper)

13

25

Reviewing a research paper: analyzing the references

It is researcher’s professional obligation to cite prior work• the manuscript being reviewed includes claims of novelties; regularly

citing prior research• the reviewer needs to check the validity of the claims• most efficient to carry out the analysis using electronic libraries

At minimum:1. Check what is found using the keywords of the article 2. Study the references you don’t know beforehand3. Check which recent papers cite the same references4. Check the references of those recent papers

26

References (start of a list of 41…)

14

27

Example (5)

Koc & Liu 1998: DCT based motion estimation

Analysis of the references:1 Check what is found using the key words of the article: not much...2 Study the key references you don’t know beforehand:

Transform domain methods: Kuglin & Hines 1975, Thomas 1987, Girod 1993 Young & Kingsbury 1993, (this reviewer would add Lees & Henshaw 1986 Proc. SPIE 730, that has its foundation in optics)

3 Check which recent papers (until 1997) cite the same referencesNo similar ideas found, just the ordinary ”whatever you do, I can do better” = perhaps 1 MPI unit contributions

4 Check the references of those recent papers5 Make your conclusions on novelty

28

15

29

A look at the most relevant references(in Young & Kingsbury 1993)

We observe the same foundation as with Koc & Liu 1998

30

Example (6)

Who has cited Kuglin & Hines 1975? (Google Scholar lists over 500 citations)Who has cited Thomas 1987? (check SCOPUS or Google Scholar)Who has cited Girod 1993? (seven references till 1997) Examples

Girod B, Steinbach E, Farber N Performance of the H.263 video compression standard J VLSI SIG PROCESS S 17 (2-3): 101-111 NOV 1997

Ku CW, Chen LG, Chen CH, et al. Investigation of a visual telephone prototyping on personal computersIEEE T CONSUM ELECTR 42 (3): 750-759 AUG 1996

Kuo CM, Hsieh CH, Jou YD, et al. Motion estimation for video compression using Kalman filteringIEEE T BROADCAST 42 (2): 110-116 JUN 1996

KOKARAM AC, MORRIS RD, FITZGERALD WJ, et al. INTERPOLATION OF MISSING DATA IN IMAGE SEQUENCES IEEE T IMAGE PROCESS 4 (11): 1509-1519 NOV 1995

………No similar contributions as in the paper we are refereeing were found………

16

31

Writing the referee reportNo fixed rules exist, the following ones are according to (Smith 1990)

• Most important: make your opinions clear; avoid ”perhaps” and ”maybe”; evaluate the paper, not the author; itemize the contributions

• Remember to give your positive comments first before the negative ones

1. State the recommendation and its justification; the five point scale part of the evaluation form is not enough

2. Show with a few summarizing sentences that you have understood the paper. The editor may use this part and compare your summary to those of the other reviewers

3. Evaluate the significance and validity of the research goal, and the noveltyof the results

4. Evaluate the quality of methodology, techniques, accuracy and presentation; recommendations for revisions can be written here

5. Make a clear recommendation for or against publication with justifications

This analysis is exactly what you need for the literature review part of your thesis!

32

Example referee report The authors propose a dual linear processor array solution for implementing xxxxxx

for xxxxx. In particular, they propose using an architecture that consists of two symmetric linear array processors for the task, and claim to cut down the computing effort. The topic is important and contributions in the area could be useful for numerous applications.

Unfortunately, the manuscript is incoherent in several respects. First, the title claims to present a xxxxx solution. However, although the architecture is capable of xxxxxxx, the authors advocate xxxxx algorithm that xxxx This is not xxxxxx, and the authors do not even cite well known and popular xxxxx techniques such as xxxxx and xxxx The technique used by the authors does not give same results with xxxxx and obviously gives inferior lower signal to noise ratio at the same bit rates.

………………..Third, the authors make a claim that their architecture can be implemented within a

moderate scale FPGA device, and in the conclusions they even make a claim “as a low power co-processor”. These claims are unsubstantiated, as no gate counts, silicon area estimates, nor power consumption estimates have been provided.

………………………….Confidential comments to the editor: There are many incoherent and incorrect claims. It is waste of time to comment them all and educate the authors who may simply integrate the observations into the next version submitted for review elsewhere.

17

33

Example of review statement on excessive self-plagiarism

The key problem with the manuscript is not about its results and methodology, but the overlap between the results presented in the conference article by the same authors titled xxxxx. This has not been cited in the manuscript, nor have any novel contributions described against the results presented in that article. Even most of the figures have been reused.

Neverthless, this reviewer sees possibilities in improving the manuscript and its contribution. The advocated idea of xxxxx is most tempting.

The actual application target is zzz for which multiple approaches exist. Prior solutions tackling the ggg problem at algorithm level include, e.g., www and qqq, however, neitherof these are cited in the current manuscript. Etc.

Recommendation: RQ - Review Again After Major Changes Comments:

34

Compiling the recommendations

Classification of papers (Smith 1990)

1. Very significant; includes major results (<1% of all papers)2. Interesting work, a good contribution (<10%)3. Minor positive contribution (10-30%)4. Elegant and technically correct, but useless5. Neither elegant nor useful, but not wrong6. Wrong and misleading7. Unreadable, impossible to evaluate

The acceptance level of the journals and conferences vary; 1,2, and perhaps 3(-4)

18

35

What to do with the referee reports on your own paper?

The quality of the referee reports vary a lot...• a good report provides additional references, and guidance to writing and

further research

If the referee report asks you to make revisions to your paper• you should make them and explain them in your letter that follows your

resubmission

If you feel dissatisfied with the quality of the referee report, especially, if it is a vague one, you may write to the editor, politely asking forfurther clarifications as the subject of researh is important to you• first, however, very carefully check the facts in the report• the editor may select an additional reviewer

Letter to the editor/anonymous referee (1)

When re-submitting after revisions, write a letter quoting the referee comments, and answering every single one, even acknowledging corrections to typographical errors

The following slides include the responses to the comments by a referee::1. “ One of the major points of the paper is that flexibility (multi-protocol, multi-standard,

etc) cost dearly in software architecture and in power consumption. The paper juxtapose this with a simpler (static) architecture of the past. The paper suggests looking into improvements in tools and technology to support better sw development. I would ask the paper writers to look into the work of static scheduling and allocation technology.”

Response: advances in static scheduling and allocation techniques are acknowledged, admitted important, and a new reference (22) has been added.

36

19

Letter to the editor/anonymous referee (2)

2. “ The abstract claims that hardware and software architectures in embedded systems are essentially the same as those in mainframe and personal computers, but the processor architectures used for embedded systems are obviously very different from superscalar out of order cores common in personal computers and the system architectures are also different (insofar as complex hardware accelerators are essential to modern mobile devices, but absent from general purpose CPUs). It seems that something else is intended by this statement, but it is not clear what.”

Response: another wording has been selected and the differences have been clarified in the abstract.

3. “In the projections of video and baseband standards, when are the "future" data points are expected to occur?”

Response: years have been added to the table. We are now witnessing developments that seem to hit right on our mark. We have added reference 23 to point out the relevant very recent results and explained their relationships to our observations.

37

Letter to the editor/anonymous referee (3)

4. “Section 2 describes video and baseband processing and mentions that they are examples of common soft and hard real-time tasks. Is this difference important to the hardware/software partitioning and energy efficiency? If so, how?”

Response: this is a very complicated issue that has now been a somewhat clarified in the end of Section 2.2: Fine grained hardware acceleration moves the applications toward hard-real time characteristics if the hardware is shared between applications.

5. “ Table 3 has separate entries for "application processors and memories" and "memories." To what does this second set of memories refer?”

Response: these are the mass memories in the devices and are now indicated in the table

38

20

Letter to the editor/anonymous referee (4)6. “The paper states that energy estimations are based off of process technology and supply

voltage. There is a mention in the abstract that these devices use low-leakage technologies, but it is not clear from the section if leakage can be and is ignored in these estimates.”

Response: we chose not to clarify this issue. In reality, leakage is included in the estimates and leakage control techniques are employed, so the issue is rather a complicated one.

7. “Ordering of entries in Figure 6 legend is confusing-- diagram would be easier to follow if ordering were by first instance (i.e. (1,12),(2,8,11),(3,4), ..)”

Response: we now explain in the text the order of entries in the legend; it is the priority order

8&9. “Page 9, the first sentence of section 3.6: "Figure 8" should read "Table 8".” “In Table 8, in the last column heading, "Degratation" should read "Degradation“”

Response: corrected

39

40

How to become a referee

Writing a publication that is cited is the most certain way to become a referee

Coordination or technical coordination of an EU RTD project is a direct road to proposal evaluations

Refereeing is very rewarding, helps to keep up-to-date andaware of developments in fields adjacent to ones own specialty

21

41

(Almost) Final Words

Good referee reports are valuable and free of charge• help in improving the paper• help in improving as a researcher• help in improving as a referee

Refereeing is a learning experience

Scientific progress rests heavily on peer reviews

42

Shortcomings of the peer review process

1. The peer review process has not been designed to detect fraud2. The peer process may take from several months to years in case of

some journals• the contributions could be outdated by the time of publication

3. The reviews can be biased, incomplete, and incorrect• sometimes a rebuttal process exists and new reviewers are

assigned to a second round of reviewing4. The reviewers are always unaccountable, frequently ignorant, and

sometimes plain wrong• open review schemes have been tried; usually no improvement in

the quality of reviews as good reviewers hard to recruit5. The means provided for dealing with review errors and injustices are

lacking6. The reviews may overlook gross errors that invalidate the results

22

43

How to avoid the reviewer’s axe (1)

Alternative guidelines for authors and (reviewers) by Stephen Senturia (Journal of Micromechanical Systems, vol. 12, no. 3, June 2003)

• (Almost) Nothing is New = ”First, figure out what you have done. Then go to the library and find it there!” ….and tell this to the reviewers

• Rely on the Believability Index= write the paper in order of decreasing believability

• Watch for Gambling Words=”obviously, probably, certainly, undoubtedly” = you don’t know what you are talking about

44

How to avoid the reviewer’s axe (2)

Guidelines by Stephen Senturia (Journal of Micromechanical Systems, vol. 12, no. 3, June 2003)

• Don’t be a Longfellow (a famous poet/storyteller)= don’t make conclusions of your results prematurely in your paper, that is, before ”Discussion”

• Don’t Pull Rabbits out of Hats= scientific articles are no detective stories

• Mine All the Gold= try to extract everything you can from your data

• Remember: Reviewers are inarticulate and authors (somewhat) paranoid~ next time write a better paper