Slide 1

Grants Factory How the peer review panel works Mick Tuite (Biosciences) BBSRC/Leverhulme Trust Simon Kirchin (SECL) AHRC Jan 2013

Slide 2

How the Peer Review Panel Works.... Q1:Who has submitted a grant? BBSRC Leverhulme Trust AHRC Q2:Who has reviewed a grant? Q3: Who has won a grant? Q4:Who has sat on a panel?

Slide 3

How the Peer Review Panel Works.... Q1:Who has submitted a grant? BBSRC Leverhulme Trust AHRC Q2:Who has reviewed a grant? Q3: Who has won a grant? Q4:Who has sat on a panel?

Slide 4

How the Peer Review Panel Works.... Q1:Who has submitted a grant? BBSRC Leverhulme Trust AHRC Q2:Who has reviewed a grant? Q3: Who has won a grant? Q4:Who has sat on a panel?

Slide 5

How the Peer Review Panel Works.... Q1:Who has submitted a grant? BBSRC Leverhulme Trust AHRC Q2:Who has reviewed a grant? Q3: Who has won a grant? Q4:Who has sat on a panel?

Slide 6

How the Peer Review Panel Works.... Submission to decision How are fund/not fund decisions reached? Scoring and ranking How to influence the panel....legally Getting on a panel Panel members what to expect....!

Slide 7

Time is at a premium for referees and panel members How much time do I have? e.g. AHRC: One day meetings 09:30 to 16:30 25 large project grants 300 minutes 12 minutes per grant! You have limited time in the frame.... BBSRC e.g. BBSRC Two day meetings 100 large project grants 600 minutes 6 minutes per grant

Slide 8

Process: from submission.....to decision Fundingagency Referees DiscussRank Internal Peer Review Panel ?

Slide 9

Composition of grant-awarding panel Academics Non-academic specialists end users Admin staff Observers Chair Panel secretary

Slide 10

Run up to panel meeting: timelines ~ 4 months: membership of panel confirmed lay summaries sent to panel members grants assigned to panel members referees identified (applicant/member/funder) potential conflicts of interest identified ~3 monthsreferees contacted ~ 3 weeks panel members receive the following: BEFORE - all grants meeting - all referees reports - all responses to referees comments - previous grant record

Slide 11

~ 4 months: membership of panel confirmed lay summaries sent to panel members grants assigned to panel members referees identified (applicant/member/funder) potential conflicts of interest identified ~ 3 monthsreferees contacted ~ 3 weeks panel members receive the following: BEFORE - all grants meeting - all referees reports - all responses to referees comments - previous grant record Run up to panel meeting: timelines

Slide 12

~ 4 months: membership of panel confirmed lay summaries sent to panel members grants assigned to panel members referees identified (applicant/member/funder) potential conflicts of interest identified ~ 3 monthsreferees contacted ~ 3 weeks panel members receive the following: BEFORE - all grants meeting - all referees reports - all responses to referees comments - previous grant record Run up to panel meeting: timelines

Slide 13

Role of the IM (Introducing Members) Two IMs per grant (one leads) One nominated to lead Almost certainly not be experts in your field Make individual recommendations (numerical score or yes/no) Open to discussion (....not always) Chair coerces consensus decision (rank/score)Introducing Member (IM)

Slide 14

Less strategic importance compared to other proposals Proposal poorly written Insufficient preliminary dataProposal lacked focus Proposal overambitious and unlikely to achieve all of its objectives Proposal lacked detail Work programmes poorly integrated Lack of relevant expertise Under resourcedPoor track record Will not significantly advance the fieldPoor value for money Will not significantly increase knowledge in the field Costs not adequately justified Is this project of international quality and therefore worthy of funding? Yes/No What are the IMs looking for? e.g. BBSRC feedback form In order of priority, please specify the STRENGTHS of the proposal: In order of priority, please specify the WEAKNESSES of the proposal:

Slide 15

Ranking grants: where is the cutoff ? Smith Jones West North South East Brown Green Black Butcher Baker.. Fund Not fund Modulators: New investigator Industrial contribution Strategic relevance Impact plans Good final report scores Strong publication record Rank order Drawing the line: Budget rules! The grey zone

Slide 16

The scoring system: panel e.g. BBSRC, Wellcome Trust Score definitions 0 = Invited resubmission 1 = Flawed 2 = Not competitive 3 = Fundable but... 4 = Good 5 = Very good 6 = Excellent 7 = Outstanding/world class 20% Discuss Not fund Fund

Slide 17

The scoring system: referees Exceptional (fundable) Excellent (fundable) Very Good (fundable) Good (fundable) Not Competitive (not fundable) Unfundable (not fundable) = in with a chance... = little chance... = not a chance... (20%) (60%) (20%) INTERNATIONAL QUALITY SCIENCE

Slide 18

Scoring impact of research Pathways to impact (PtI) document: What will be done to ensure that potential beneficiaries have the opportunity to engage with this research? SCORING SYSTEM Excellent Satisfactory Unsatisfactory How will the impact scores be used? -Taken into account when rank ordering - Feedback to applicants - Not used to reject outstanding applications

Slide 19

Influencing the panel....legally.... Profile they know who you are Research/conference papers Seminars/conferences Presentation of grant 12pt Arial (11pt allowed), layout Use of images (unpublished data) Proofread! Avoid extensive self-citation Lay/technical summary All elements of the form are important Response to referees Succinct, courteous Include new information

Slide 20

Influencing the panel....legally.... Profile they know who you are Research/conference papers Seminars/conferences Presentation of grant 12pt Arial (11pt allowed), layout Use of images (unpublished data) Proofread! Avoid extensive self-citation Lay/technical summary All elements of the form are important Response to referees Succinct, courteous Include new information

Slide 21

Lay/technical summary What is it? Lay: Describe the proposed research in simple terms in a way that could be publicised to a general audience [up to 4000 chars] Technical: Describe the proposed research in a manner suitable for a specialist reader. [up to 2000 characters] Why is it important? Read by all members of the panel Can attract non-IMs to take an interest in your grant What it should do? Make it clear - why this project is interesting - why the project must be funded - what the wider impact of the project will be

Slide 22

How do you get on a panel ? Good funding record Good refereeing record on time sensible comments Personal contacts Personal contacts panel members staff of funding agency A pply!

Slide 23

Outline Stage Reviewed by 1 member of the Advisory Panel (35 academics) Decision either full application or reject Member suggests four referees if recommend full application Decision sent to the Director who approves or rejects them Full application stage Sent to 4 referees, 2 suggested by you and 2 selected by Trust member (final choice Not made by Advisory Panel member) Based on the reviews the Trustees * make the final decision. Leverhulme Trust: How grants are assessed * Footnote: Trustees are mainly current/retired Directors of Unilever

Slide 24

The panel memberswho are they? What you hope for..... Informed Informed knows your research area No bias No bias institution or research area Can identify strengths Can identify strengths (but not so good on weaknesses...?) Listens Listens to others and takes on board their comments Makes clear decisions/recommendations Makes clear decisions/recommendations Dr Perfect

Slide 25

Dr Know it all Has an opinion on all grants Seems to know a lot....but actually knows very little Dr On the ball Everyone listens to what she/he says Actually does know a lot about a lot...and everyone agrees with his/her decision Dr Methodical Has read all the grants (not just case for support) Can recall fine detail e.g. year applicant got PhD....but unable to make an informed decision.... The panel memberswhat you get!!

Slide 26

Dr Elitist Thinks all projects fundable....provided the PI is from.....Oxford.....Cambridge, or.....Imperial Dr Its all crap Finds nothing of value in any grant Makes rude comments about PI........and no faith in referees integrity Dr Confused by it all Struggles to get to grips with any of the grants Has more questions than opinions....easily persuaded by others ?

Slide 27

Dr Disorganised Still reading his/her grants during the meeting Left key (slanderous) referees comments on train Chaotic presentation as IM Dr Gullable Believes that.......the applicant knows what he/she is talking about....the referees are always right....the chair (or anyone else) can override his/her decision The Clown Little input to discussion other than jokes Can see the funny side of referees comments Liked by everyone...comments ignored...

Slide 28

Being on a grants panelwhat to expect 1. Hard work: 4-20 grants per panel meeting 2. You get paid! 3. What makes a grant fundable 4. Disillusioned: many good grants are not funded 5.Exposure to a wide range of topics/fields 6.....confidence in the peer review system

Slide 29

The key take home points. 1. You have little time to make an impact! 2. Get your work known by panel members 3. Pay particular attention to the lay and technical summaries 4. Do not underestimate the role/importance of the panel members 5. Not all grants in the funding zone on the day... are actually funded 6. If you really want to know what goes on get yourself onto a panel!

Slide 30

Some points for discussion. Q: Is this a fair and transparent process? Q:What are the alternatives? - hypothetical.... - real... Q: Do we need referees and panels? Q:What value the internal peer review system? Q. Any funder-specific questions?