review process - how to reviewdisi.unitn.it/~tomasi/think/6-review.pdf · 1) presentations...

Fausto Giunchiglia 2005 Hannes Werthner 2003 1

Introduction to Science and its methods

Review Process - How to review

Fausto Giunchiglia

By Fausto Giunchiglia and Alessandro Tomasi

Index:

1. Review Form 12. Review Form 23. Answer to the Reviews4. Review Process



1. Review Form 1

1. Review Form 1

REVIEWER CODE:

1) Title

2) Author(s)

3) Paper Summary [short description of the message and maybe of how it has been developed. What if more than one message or no message?]



1. Review Form 1

4) Type of Paper/Research described[Chose one of the options below and explain]

- Research (with original results) of which kind (Theory, expemerimental, ...)

- Application (case study, ...)- Synthesis of recent advances- Other. Please specify

1. Review Form 1

5) General Ratings[Rate within Bad/Weak/Fair/Good/Excellent] [0/1/2/3/4/5]

Put the score and some text motivating your score about:5a) Relevance (with respect to the reference community) 5b) Originality (incremental, new work, ...)5c) Significance of the work (how big the gap from the state of the art) 3d) Technical soundness 3e) References3f) Presentation



1. Review Form 1

6) Technical Soundness[Chose one of the options below and some text motivating the choice]

- Technically correct- Minor errors (indicate them)- Major errors (indicate them)- Unsupported claims (provide a detailed explanation)

1. Review Form 1

7) Presentation [Rate within Yes/Somewhat/No]

Put the score and some text motivating your score7a) Are the title and abstract appropriate?7b) Is the paper well-organized (discuss course and fine grained structure)?7c) Is the paper easy to read and understand?7d) Are figures/tables/illustrations sufficient?7e) Is the English acceptable? 7f) Is the paper free of typographical/grammatical errors?7g) Is the reference section complete?



1. Review Form 1

8) General Recommendation[Please, chose one of the options below]

- Very strong accept (beautiful paper!)- Strong accept (excellent and important contribution)- Weak accept (good paper, some new interesting ideas)- Weak reject (marginal, weak content, would require a major revision)- Strong reject (unreadable, nothing new, premature, contains major errors)

1. Review Form 1

9) Main Reason for your Decision[For accept choices please indicate one of the options below] - accept because of the originality (good ideas, sound presentation)- accept because of the quality of the proposed synthesis (useful review

on recent advances)- other

[For reject choices please indicate one of the options below] - reject because it is not relevant for the conference- reject because of the presentation (unreadable, unstructured)- reject because the content is too premature for really making sense- reject because of the lack of originality (results already known, or similar

overview already published)- reject because of major errors



1. Review Form 1

10) Your Level of Expertise (Compared to Level of Others)- I am an expert of the field and know the relevant literature- I understand the problem, I know some of the state of the art- I only have a superficial understandings of the issues

11) Does the paper qualify for the best paper award [Y/N]

12) Comments to the Author(s)[Please, provide here a clear justification of your ratings, in particular with regards to the overall recommendation]

13) Additional Comments to the Author(after circulation of reviews among reviewers) (Can be empty, cannot change previous review)

14) Additional Comments as Answer to Author's Answer and Modifications(Can Be Very Short:

- Evaluation Of Author Answer- Value Judgement- Final Score (Possibly Changed))

1. Review Form 1



2. Review Form 2

2. Review Form 2

REVIEWER CODE:

1) Title

2) Author(s)



2. Review Form 2

3) Main Message:

Relevance: How relevant is the paper to the workshop? 0: not relevant at all1: rather not relevant2: relevant3: very relevant

Technical Quality: What is the technical quality of the paper? 0: really bad1: bad2: good3: really good

2. Review Form 2

Presentation: What is the overall presentation of the paper? 0: really bad1: bad2: good3: really good

Overall Ranking: What is your overall recommendation? 0: strong reject1: reject2: weak reject3: weak accept4: accept5: strong accept



2. Review Form 2

Confidence: Reviewer's expertise in the area0: I know little about this area1: I know enough about this area2: I have good expertise in this area

Why to accept? What are the most important reasons to accept this paper? (1-3 sentences)

Why to not accept? What are the most important reasons NOT to accept this paper? (1-3 sentences)

Comments: Detailed comments on the paper (primarily for the authors)

3. Answer to the Reviews



<Brief introduction>LIST OF {<general comment quoted from reviews><your answer arguing how you have accordingly modified the paper>}

Moving now to the more specific comments:LIST OF {<specific comment quoted from reviews><your answer arguing how you have accordingly modified the paper, providing detail but not too much>}

<Concluding sentence>

3. Answer to the Reviews

4. Review Process



4. Review Process

0) Abstract(send it to [email protected] not later than a week before the presentation)1) Presentations(Fortunate situation where you may know of what the paper is about)2) All Papers Submitted by June, 13th3) Review Allocation (by Program Chair) by June, 20th4) Reviews Due by July, 4th5) Reviews circulated to Reviewers for additional comments6) Reviews send to Author by July, 11th7) Author sends back answer and modified paper

by July, 23rd8) Reviewers provide final answer 9) Exam Pass/No Pass (of Authors and Reviewers)

by July, 29th

4. Notice!!!

In the paper you have to use the same version of the stylefiles. It is available by downloading it fromhttp://www.springeronline.com/sgw/cda/frontpage/0,11855,3-111-2-124365-0,00.html

Avoid using stylefiles acquired from other sources as these may not be correct



How to dopresentations

1. Presentation methods

2. Attitude towards the audience

3. How to select content

4. How to structure

5. Introduction

6. Stage fright

7. Main part

Structure



8. Using pictures

9. Visual aids

10. Interposed questions, interruptions

11. Final discussion

12. The end

13. Optical – acoustic factors

14. Ancillary conditions

Structure

1. Presentation methods

Rule No. 1:Control effect

Rule No. 2:Integrate audience



1. Presentationmethods

• Seminar

• Lecture

• Presentation with final discussion

• Presentation with intermediate

discussion

Comparison of somepresentation methods

What remainsTime to

prepare / h

80% 4,0

Presentation with Intermediate discussion

60% 2,5

30% 1,7

Lecture 20% 1,0

Open discussion

Presentation with Final discussion



2. Attitude towards theaudience

• Who is the audience

• What do they know

• What are they interested in

• What do they understand

2. Attitude towards theaudience

• Contact with eye• Simple and understandable

languages• Concentrate on the important• Credibility • Reply to objections• Avoid „techno“ language



How much?From where?What is interesting?What is the

objective?What should the

audience do?

Do not want toomuch!

3. How to select content

4. How to structure– catch words

• Speak as you think –follow the thinking

• Use written concept of catch words



5. Introduction

• Try to find common understandingwith the audience

• Lead to the subject• Initially: create attention

Say at the beginning• The structure• The length• Invite to a dialogue

But at the beginning:Who are you

5. Introduction



attention

Length of presentation

Hope thatit will end soon

6. Stage-fright

• Is natural!• Everybody has it !• Is not forever!

Afraid to speak



Reduce Stage-fright

• Be well prepared!

• Learn important parts by hard!

• Relax!

• Look for a positive „point“

• Do some contacts before presentation!

Stucked?



7. Main part

Introduction

Main part• Current situation• Potential solution• How to realise

End

8. Using pictures / figures

• Take from the context of the audience• Has to increase the message• To explain the issue• Does not be an end on itself

Picture



8. Metaphoricallanguage

• To be a comparison• Should be practial• Story• Citation• Joke

9. Visual aids

• Black board• Projector• Film• Videobeam• Overhead-Display



? GraficalLayout?Colours – Contrast – Background?Clipart-files?Fonts (size)

? Produce hand outs

Presentation programs

What we keep

10% Reading20% Hearing30% Seeing50% Hearing &

Seeing70% Reporting 90% Do it yourself



10. Questions

• Question

• Are welcome• Should be answered

immediatly• Bit can also be delayed or

forwarded to somebodyelse

• Interupptions

• ignore!

11. Final discussion

• Announce already in the introduction• Write up issues of discussion• Keep sequence of questions• Repeat questions (if necessary)• Draw conclusions



12. End

•Avoid• New questions which do not help in the conclusions

•Main ideas• Should be summarized

12. End

Main idea + review objective + how to realise=

Good end



13. Acoustic factors

• No „speaking smoking"• Speak loudly and slowly• You should vary volume and speed• Make breaks• Try presentation beforehand

• With your body• With your hands• With your eyes

13. Optical factors

Behave natural!



14. Othercircumstanceas

• Room size

• Lighting

• Ventilation

• Sockets

• Other means (Microfon)

• Breaks

Reading - How to read

Fausto Giunchiglia

Literature:Bruno Buchberger, Thinking Speaking Writing

By Fausto Giunchiglia and Alessandro Tomasi



Index:

1. The Role of Literature2. What is literature

2.1 Classify the results3. The Documentation of Literature4. Steps in the Use of Literature

1. The Role of Literature



“Keep Re-Inventing the wheel”


For many problems:

• the solutions is already in the literature

• the literature has solutions in similar problems


Necessary

Useful

but not

Necessary

It is necessary to know

how the literature is organized:

• within your research community

• outside your research community

[translate other concepts

in our community]




There is an interleaving of:

1) Try to solve the problem yourself

2) Read the Literature


Interactive Process of Reading Papers:

Step 1

Step 2

Step 3

…

Quantity

Depth




How do I know when I can stop reading?

When I know the 90% of the paperscited by relevant papers


To keep Scholarship:

• go to conference

• talk to people (best opportunity in conference)

• journals

• proceedings

• web

• …



2. What is Literature


More importantthan papers

Main Wayto do Career

1. Monographs / Books

2. Journal Articles

3. Articles in Collections

4. Papers in Proceedings of Conference

5. Papers in Proceedings of Workshop

6. Technical Reports

7. Grey Literature (Ex: Web Sites)



Each research community has:

1-2 Top Journals

1-2 Top Conference


2.1 Classify the results




1. Monographs / Books:

Contents:extensive/complete description of research

Originality:None (2/3 years old)

Note:the research is finished

Quality Control:Very Good


2. Journal Articles:

Contents:Very good, very crisp, specialized

Originality:Full of two years ago (in CS is obsolete)

Note:Come regularly

Quality Control:Very High (Many shots)




3. Articles in Collections:

Contents:Very good, very crisp, specialized

Originality:Full of two years ago (in CS is obsolete)

Note:Not regularly

Quality Control:Lower


4. Conferences:

Contents:New results, almost complete, semi-final

Originality:Good (6 months ago)

Note:Is Refereed

Quality Control:Medium (only one shot: accept or reject)




5. Workshop

Contents:New results, almost complete, semi-final

Originality:Good (6 months ago)

Note:Is Not Refereed

Quality Control:Medium (only one shot: accept or reject)


6. Technical Reports

Contents:Detailed about a specific topic

Originality:Maybe

Note:---

Quality Control:Low (No Refereeing Process)




7. Grey Literature (Ex: Web Sites)

Contents:Random

Originality:Random

Note:The main way to publish our results

Quality Control:Random

3. The Documentation of Literature




• Library- author files- keywords files

• Review Journals- Computing Journal- Artificial Intelligence Review- Scientific Magazines- Scientific Magazines for Spreading Communications- Journal for Publish new Results

• Bibliographies• Citation index• Research index


Research Community are organizedinto societies

Produce:Organized Conferences, Journals, Magazines

Examples of Societies:ACM, IEEE, ECCAI, VLDB, …




Citations Index:

• defines journals

• cites titles, authors, abstract of all paper

Example:ISI, the most important for Science


Research Index:

• the new game

• is on line

• how many times you are cited in Internet

Example:

http://citeseer.nj.nec.com



4. Steps in the Use of Literature

4. Steps in the Use of Literature

1. Use CiteSeer (http://citeseer.nj.nec.com)

2. Ask to Colleagues, Advisor,

Friends, … by e-Mail

3. Consult References

4. Consult Review Journals

5. Ask Authors

review process - how to reviewdisi.unitn.it/~tomasi/think/6-review.pdf · 1) presentations...

Documents