Societal Challenge 6: Europe in a changing world - Inclusive, Innovative and Reflective Societies : 2017 Calls : Single stage procedure :

• - Co-creation for growth and inclusion

• - Understanding Europe Promoting the European public and cultural space - Reversing inequalities and promoting fairness

• - Engaging together globally

Expert's Briefing

26.01.2017

Margaux Genachte– Call coordinator

REA.B3. Societal Challenge 6: Europe in a changing world - Inclusive, Innovative and Reflective Societies

HORIZON 2020 PROPOSAL EVALUATION

HORIZON 2020

2

Agenda 1. Role of independent experts

2. The evaluation procedure in practice

3. The evaluation criteria

4. Schedule

5. Logistics

HORIZON 2020 PROPOSAL EVALUATION

1. ROLE OF INDEPENDENT EXPERTS

HORIZON 2020

4

Role of independent experts

As an independent expert, you evaluate proposals submitted in response to a given call

You are responsible for carrying out the evaluation of the proposals yourself

− You are not allowed to delegate the work to another person!

You must close reports in the electronic system within a given deadline

− This is part of your contractual obligations!

− The allowance/expenses you claim may be reduced or rejected otherwise

Significant funding decisions will be made on the basis of your assessment

If you suspect any form of misconduct (e.g. plagiarism, double funding), please report this to REA staff

HORIZON 2020

5

Guiding principles

Independence

− You are evaluating in a personal capacity − You represent neither your employer, nor your country!

Impartiality

− You must treat all proposals equally and evaluate them impartially on their merits, irrespective of their origin or the identity of applicants

Objectivity

− You evaluate each proposal as submitted; meaning on its own merit, not its potential if certain changes were to be made

Accuracy

− You make your judgment against the official evaluation criteria and the call or topic the proposal addresses, and nothing else

Consistency

− You apply the same standard of judgment to all proposals

HORIZON 2020

6

Confidentiality

You must:

Not discuss evaluation matters, such as the content of proposals, the evaluation results or the opinions of fellow experts, with anyone, including:

− Other experts or Commission/Agencies staff or any other person (e.g. colleagues, students…) not directly involved in the evaluation of the proposal

− The sole exception: your fellow experts who are evaluating the same proposal in a consensus group or panel review

Not contact partners in the consortium, sub-contractors or any third parties

Not disclose the names of your fellow experts

− The Commission publishes the names of the experts annually - as a group, no link can be made between an expert and a proposal

Maintain the confidentiality of documents, paper or electronic, at all

times and wherever you do your evaluation work (on-site or remotely)

− Please take nothing away from the evaluation building (be it paper or electronic) − Return, destroy or delete all confidential documents, paper or electronic, upon

completing your work, as instructed

HORIZON 2020

7

Conflict of Interest (COI)

You have a COI if you:

• were involved in the preparation of the proposal (including pre-proposal checks/'mock' evaluations)

• stand to benefit directly/indirectly if the proposal is successful

• have a close family/personal relationship with any person representing an applicant legal entity

• are a director/trustee/partner of an applicant or involved in the management of an applicant's organisation

• are employed or contracted by an applicant or a named subcontractor

• are a member of an Advisory Group or Programme Committee in an area related to the call in question

• are a National Contact Point or are directly working for the Enterprise Europe Network

HORIZON 2020

8

In the following situations, the REA will decide whether a COI exists:

− Were employed by an applicant or sub-contractor in the last 3 years

− Were involved in a grant agreement/decision, the membership of management structures or a research collaboration with an applicant in the last 3 years

− Are in any other situation that casts doubt on your impartiality or that could reasonably appear to do so

e.g. you are involved in a competing proposal

COI conditions are spelled out in your contract in the Code of Conduct (Annex 1)

Conflict of Interest (COI)

HORIZON 2020

9

You must inform the REA as soon as you become aware of a COI

− Before the signature of the contract

− Upon receipt of proposals, or

− During the course of your work

If there is a COI for a certain proposal you cannot evaluate it

− Neither individually

− Nor in the consensus group

− Nor in the panel review

− The REA will determine if there is a COI on a case-by-case basis and decide the course of action to follow

If you knowingly hide a COI, you will be excluded from the evaluation and your work declared null and void

− The allowance/expenses you claimed may be reduced, rejected or recovered

− Your contract may be terminated

Conflict of Interest (COI)

HORIZON 2020 PROPOSAL EVALUATION

2. THE EVALUATION PROCEDURE IN PRACTICE

Single Stage - calls 14 Topics delegated to the REA:

CO-CREATION: topics 1, 4, 6, 7 & 8

REV-INEQUAL: topic 13

CULT-COOP: topic 11

ENG-GLOBALLY: topics 1, 2, 3, 5, 6, 8 & 10

Evaluation modalities :

FULL REMOTE EVALUATION – no CENTRAL

Consensus will need to be reached on 100% of the proposals evaluated

Only panels will be organized in Brussels

Total Budget of the 14 topics: 71,55 million

Single stage calls Time to inform (TTI) : 30 May 2017

Single stage calls Time to grant (TTG): 04 October 2017

Main list: ± 25 projects:

SC6 – 2017 Single stage calls and

CO-CREATION REV-INEQUAL CULT-COOP 11 ENG-GLOB

12 1 2 10

HORIZON 2020

12

FULL Remote Evaluation (individual assessment & remote consensus on 100% of the proposals):

Each proposal is assessed independently and individually by 3 experts against 3 pre-determined evaluation criteria

Individual opinions are recorded in an Individual Evaluation Report (IER), giving scores and comments against the evaluation criteria

Once all IERs are finalized, one of the 3 experts acts as rapporteur and drafts the Consensus Report (CR)

If no consensus can be reached remotely, a minority/majority opinion is recorded in the CR and the proposal will be discussed and decided in the panel

HORIZON 2020

13

Roles: Who does what ? Evaluators

• Evaluate proposals

• Submit Individual Evaluation Reports (IERs) remotely

• Participate in remote consensus discussions to agree on scores and comments

• Participate to panels meeting in Brussels ( 5 evaluators per panels)

Vice-chairs

• Do not evaluate proposals

• Quality check of Individual and Consensus Reports

• Support REA project Officers in monitoring evaluation progress and deadlines

Observers

• Observe the evaluation process

• Provide advice / improvement suggestions to the REA

Contact persons:

• Primarily: your REA topic leader

• Call coordination: Margaux Genachte and Amelie Birot-Courcy

HORIZON 2020

14

Vice Chairs (VC)

Vice Chairs are experts who assist REA officers in order to ensure that the evaluation exercise runs in a smooth way.

VCs do not evaluate proposals, and cannot give their opinion on any proposals.

Their contribution relates to support regarding the evaluation management

HORIZON 2020

15

Vice Chairs (VC) -2-

VC will contact experts at the start of the evaluation and will be in contact through the entire remote evaluation exercise. VCs ensure a quality and timely delivery of evaluators' reports – Individual Evaluation Reports (IERs) and Consensus Reports (CRs)

VCs provide guidance and feedback to evaluators, under the control and overall supervision of REA topic leaders. Please be receptive to their comments and guidance.

VCs are in constant touch with REA staff to raise any relevant issue or report on any necessary event.

HORIZON 2020

16

Read the proposal and evaluate it against the evaluation criteria

− Without discussing it with anybody else

− As submitted - not on its potential if certain changes were to be made

Disregard excess pages marked with a watermark

Give the benefit of the doubt if the proposal contains contradictory information

− Inform the call coordinator if you consider that the proposal contains 'obvious clerical errors' or the proposal is incomplete

Individual evaluation

HORIZON 2020

17

Complete an Individual Evaluation Report (IER)

− Give comments on each evaluation sub-criterion in terms of strengths and weaknesses of the proposal

− This is helpful for rapporteur if statements are qualified with examples

− Give score matching comments

− Explain shortcomings, but do not make recommendations

− Give your view on operational capacity of partners

− Assess role of third countries/international organisations and indicate whether or not they should be exceptionally funded

− Sign and submit the form in the electronic system ( SEP)

Individual evaluation

HORIZON 2020

18

Drafting the IER

X Poor comments merely echo the score

a Good comments explain it

X Poor comments are ambiguous

a Good comments are clear

X Poor comments are vague, subject to interpretation

a Good comments are precise and final

X Poor comments provide an opening for a complaint

a Good comments close the question

HORIZON 2020

19

Why say poor when you can say words like….

Insufficient, minimal, fails to describe, inadequate, very generic, not evident,

unfocused, very weak, does not meet requirements, no sufficient information,

inappropriate concerning X, limited, unclear, not sound enough, not specified,

no significant impact, not been followed, unjustified, overestimated, does not

fit profile…..

Why say excellent when you can say words like…

Extremely relevant, credible, very clear, precisely specified, realistic, very

innovative, well suited, timely, convincing, comprehensive, high quality,

justified, very well identified, strong, highly effective, thoughtful, very

promising, evidence, well-formulated, carefully-prepared, very professionally

prepared, fully in line, sound, very convincingly integrated, clearly articulated,

coherent, well balanced, ambitious, clear advances, ……

Drafting the IER

HORIZON 2020

20

Vice Chair (during Individual evaluation )

Your 1st IER will be checked by a Vice Chair ( Recorded as Quality Checker in SEP IT tool)

(do not submit at this stage i.e. save it and inform your Vice Chair)

• all sub-criteria have been addressed and include the correct information

• the IER does not summarise or describe the proposal • the IER contains statements of assessment and examples

and justifications are included to back up these assessments

• there are no recommendations, if there are, they should be rephrased as weaknesses and the score should be reduced

• the scores match the comments

HORIZON 2020

21

Remote drafting of CR by Rapporteur

Rapporteur's task

• The rapporteur is one of the evaluators.

• He/She will prepare draft CRs:

• 'Initialise' the draft CR in SEP

• Identify points of convergence and propose a consensus wording

• Identify divergences as points for discussion or clarification with other experts (via SEP's task comments box), highlighting strengths and weaknesses of proposals to guide remote consensus and potential scores

HORIZON 2020

22

Consensus - Remote only

The rapporteur is responsible for drafting the consensus report (CR)

− The draft CR is based on the comments and scores of all the evaluators as expressed in the Individual Evaluation Reports.

• The aim is to find agreement on comments and scores

Use the electronic system for evaluation (comment box) to discuss when needed.

- Agree comments before scores!

• “Outlying” opinions need to be explored

- They might be as valid as others – be open-minded

• The draft CR should be commented by all experts after it is reviewed by the vice-chair

• To finalize the CR, all experts have to approve it in the system.

• If no consensus can be reached remotely, additional experts will be added. If there is still disagreement and no consensus can be reached, a minority/majority opinion is recorded in the CR and the proposal will be discussed and decided in the panel

HORIZON 2020

23

Panel REVIEW MEETINGS, Tasks : ▪ Ensure the consistency of comments and scores.

▪ Resolve cases where a minority view is recorded.

▪ Resolve the cases in which the following issues have been flagged: operational capacity of the applicants, out of scope proposals, third country participation.

▪ Address the specific cases of resubmitted proposals (if any).

▪ Endorse the final scores and comments for each proposal (any new comments and scores -if necessary- should be carefully justified).

▪ Prioritise proposals with identical total scores, after any adjustments for consistency.

▪ Recommend a ranked list of proposals for funding (in priority order).

▪ 5 experts per topic will be selected to attend the panels with the help of VCs and POs; they will be informed during the remote phase

HORIZON 2020

27

Observer

• Appointed by the REA, may attend any meetings or monitor remote evaluation, to ensure a high quality evaluation

• Checks the functioning and running of the overall process

• Advises, in the Observer report, on the conduct and fairness of the evaluation sessions and, if necessary, suggests possible improvements

• Does not evaluate proposals and, therefore, does not express any opinion on their quality

• May raise any questions - please give your full support

HORIZON 2020 PROPOSAL EVALUATION

3. THE EVALUATION CRITERIA

HORIZON 2020

29

PROPOSAL STRUCTURE / ACTIONS / AWARD CRITERIA

Proposals will be evaluated on the basis of the award criteria 'excellence', 'impact' and 'quality and efficiency of the implementation'.

Each evaluation criterion refers to a specific SECTION of the PROPOSAL where applicants need to demonstrate that they have adequately addressed the criterion in question.

HORIZON 2020

30

Structure of the Proposal Part A :

− General Information – Abstract

− Participants and contacts

− Budget

− Ethics

− Call specific question – Open Research Data Pilot

Part B: is divided in 5 section, each award criterion refers to a specific section of the proposal where the applicants need to demonstrate that they have adequately addressed the criterion in question. If they havn't, this must be reflected in a lower score.

− Section 1: Excellence (objectives; relation to WP; concept & approach; ambition)

− Section 2: Impact (expected impacts; measures to maximize impact which include dissemination & exploitation of results and communication activities)

− Section 3: Implementation (work plan; management structure & procedures; consortium; resources)

− Section 4: Members of the consortium

− Section 5: Ethics and security

HORIZON 2020

31

Type of Actions Projects funded under Horizon 2020 can take different pre-defined shapes, in terms of requirements on partnership (mono- or multipartner), purpose (research, innovation, networking, fellowship ...), reimbursement rate etc. These pre-defined project shapes are referred to as "types of action". The SC6 calls are addressed by 3 types actions depending of the scope of the topic. However, the 14 topics that will be evaluated during this exercise will only include 2 types of actions: a. Research and innovation action (RIA) – Action consisting of activities aiming at establishing new knowledge and/or to explore the feasibility of a new or improved technology, product, process, service or solution. For this purpose, they may include basic and applied research but limited demonstration or pilot activities aiming to show the technical feasibility in a near operational environment. b. Innovation Action (IA) – Action primarily consisting of activities directly aiming at producing plans and arrangements or designs for new, altered or improved products, processes or services. For this purpose they may include prototyping, testing, demonstrating, piloting, large-scale but limited research and development activities.

Award criteria

Clarity and pertinence of the objectives

Soundness of the concept, and credibility of the proposed methodology

Extent that proposed work is beyond the state of the art, and demonstrates innovation potential (e.g. ground-breaking objectives, novel concepts and approaches, new products, services or business and organizational models)

Appropriate consideration of interdisciplinary approaches and, where relevant, use of stakeholder knowledge.

Excellen

ce

The expected impacts listed in the work programme under the relevant topic

Any substantial impacts not mentioned in the WP, that would enhance innovation capacity; create new market opportunities, strengthen competitiveness and growth of companies, address issues related to climate change or the environment, or bring other important benefits for society

Quality of proposed measures to exploit and disseminate project results (including IPR, manage data research where relevant);communicate the project activities to different target audiences

Im

pact

Quality and effectiveness of the work plan, including extent to which resources assigned in work packages are in line with objectives/deliverables

Appropriateness of management structures and procedures, including risk and innovation management

Complementarity of the participants which the consortium as a whole brings together expertise

Appropriateness of allocation of tasks, ensuring that all participants have a valid role and adequate resources in the project to fulfill that role

Im

ple

men

tati

on

IA -RIA

HORIZON 2020

33

Proposal scoring Evaluation scores will be awarded for the criteria, and not for sub-criteria listed under each criterion.

For each proposal, you give a score of between 0 and 5 to each criterion based on your comments

− The whole range of scores should be used − Use steps of 0.5 − Scores must pass thresholds if a proposal is to be

considered for funding

Thresholds for individual criteria 3

The overall threshold, applying to the sum of the three individual scores, will be 10.

HORIZON 2020

34

What do the scores mean: 0 - The proposal fails to address the criterion or cannot be assessed due to missing or incomplete information (unless the result of an ‘obvious clerical error’)

1 - Poor: the criterion is inadequately addressed or there are serious inherent weaknesses

2 - Fair: the proposal broadly addresses the criterion but there are significant weaknesses

3 - Good: the proposal addresses the criterion well but with a number of shortcomings

4 - Very good: the proposal addresses the criterion very well but with a small number of shortcomings

5 - Excellent: the proposal successfully addresses all relevant aspects of the criterion; any shortcomings are minor.

Half-marks can be used.

For Innovation actions, to determine the ranking, the score for the criterion ‘impact’ will be given a weight of 1.5.

Admissibility:

− Admissibility is checked by the Agency

− Admissibility important elements: (1) Inclusion of enough data to assess operational capacity; (2) Inclusion of a plan for exploitation and dissemination of results

− If you spot an issue relating to admissibility, please inform the Agency, especially on the above 2 points

Eligibility:

− A proposal is eligible if it complies to the topic description and the minimum number of partners as set out in the call conditions

− The conditions are checked by the agency but you are responsible to check if the content of a proposal corresponds, wholly or in part, to the description

of the call or topic (if it doesn't, proposal should be declared out of scope)

− If you spot an issue relating to eligibility, please inform the Agency immediately

− A proposal will only be deemed ineligible/inadmissible in clear-cut cases

EvalAdmissibility & eligibility checks

Page limits: Clearly set out in electronic system; excess

page(s) marked with a watermark

HORIZON 2020

36

Operational capacity

As part of the Individual Evaluation, give your view on whether each applicant has the necessary basic operational capacity to carry out their proposed activity(ies)

− CV or description of profile of persons primarily responsible for carrying out proposed activities

− Relevant publications, products, services or other achievements

− Relevant previous projects or activities

− Description of any significant infrastructure or any major items of technical equipment relevant to proposed work

Source for check: parts A & B of the proposal and/or external sources

Section on Operational Capacity to be completed in the IER and CR

HORIZON 2020

37

Operational capacity

• If at the individual evaluation stage, you consider that a partner lacks basic operational capacity, continue to evaluate full proposal including parts related to partner concerned

• If at the consensus stage, experts agree that partner lacks basic operational capacity, you evaluate the proposal without taking into account the partner concerned and its associated activities

• This check is primarily designed to identify manifestly inadequate (or even fraudulent) partners

HORIZON 2020

38

3rd countries & International Orgs

• Legal entities from Associated Countries (AC) can participate under the same conditions as legal entities from Member States

• Third countries are those which are not a Member State or AC

• Participation of third countries and International Organisations (IO) is welcome. However, funding will be subject to existing agreement

• Applicants from third countries and IO that do not automatically receive EU funding may exceptionally be granted funding if their participation has clear benefits for the consortium

If your view is that they should not receive exceptional funding, the proposal should not be penalised, except if the particular organisation is a dominant partner

As of 01 January 2017, the following countries are associated to Horizon 2020:

Associated Countries: Albania, Armenia, Bosnia and

Herzegovina, Faroe Islands, Iceland, Israel, Moldova, Montenegro, Norway, FYROM, Tunisia, Turkey, Serbia,

Ukraine, Georgia, Swisterland

Additional cSCOPE

riteria

Before evaluating a proposal, experts need to assess whether the proposal is in scope or not, according to the scope defined in the Work Program for the topic

All experts need to agree

If the proposal is only partially in scope, they need to evaluate

If the proposals is completely out of scope, it will be declared ineligible

Role of vice chairs: FLAG ALL CASES TO PROJECT OFFICERS

HORIZON 2020

40

Cross-cutting issues Cross-cutting issues are fully integrated in the work

programme (WP):

Social Sciences and Humanities (SSH)

Gender dimension in the content of R&I which addresses the relevance of sex/gender analysis, is flagged in some topics

The new strategic approach to international cooperation

Open Innovation:

• If cross-cutting issues are explicitly mentioned in the scope of the call/topic and not properly addressed (or their non-relevance justified), you must reflect this in a lower score for the relevant criterion.

ng issues

Topic Gender SSH International cooperation

CO-CREATION-1 X X

CO-CREATION-4

CO-CREATION-6 X

CO-CREATION-7 X X

CO-CREATION-8 X

REV-INEQUAL-13 X X X

ENG-GLOBALLY-1 X X

ENG-GLOBALLY-2 X X

ENG-GLOBALLY-3 X X X

ENG-GLOBALLY-5 X X

ENG-GLOBALLY-6 X X

ENG-GLOBALLY-10 X X

ENG-GLOBALLY-8 X X

Cross-cutting issues

Topic Gender SSH International cooperation

CULT-COOP-11 X

HORIZON 2020 PROPOSAL EVALUATION

4. SCHEDULE

REMOTE EVALUATION

•Individual assessment

•Full remote 100 % consensus

CR/ESR Finalisation – Cross-reading

•Experts invited to panels will cross-read the proposals to be discussed

PANEL MEETINGS

•Final ranked list by topic/call

Call deadline 02 February 2017

27 February -27 March 28 March-28 April

EVALUATION

Info to applicants: End May 2017 –beginning June 2017.

2 May–8 May

HORIZON 2020

45

SCHELDULE

Remote Evaluation :

1: Remote of 4 weeks: 27/02-27/03 .

You will have 4 intense weeks to perform an individual assessment of all proposals assigned to you and then find a consensus on 100% of them. 2 : Cross-reading, finalization of ESRs, preparation of panel meetings: 28/03-28/04/2017 ( only experts invited to panels)

CENTRAL PHASE :

PANELS : O2 – 08 May 2017 in Brussel ( 5 experts invited by topics)

HORIZON 2020 PROPOSAL EVALUATION

5. LOGISTICS

HORIZON 2020

47

Logistics

• The electronic system (SEP) for the evaluation of proposals is available and accessible via your ECAS password

− Please make sure you know your ECAS login and password

You can access through the Participant Portal:

https://ec.europa.eu/research/participants/evaluation/

→ Check the briefing on SEP workflow for further details

HORIZON 2020

48

THANK YOU VERY MUCH FOR YOUR

ATTENTION