Horizon 2020 Proposal

EvaluationDr Alex Berry

UKRO European Advisor

13 December 2016

Alexandra.berry@bbsrc.ac.uk

• Horizon 2020 evaluation process general overview– How to become an evaluator

– Evaluation criteria

• ERC evaluation process

• Marie Skłodowska-Curie Actions evaluation process

• Further information

Content

Horizon 2020:

EvaluationProcess overview

Overview of the process

Receipt of proposals

Individualevaluation

Consensusgroup

Panel Review Finalisation

Evaluators

IndividualEvaluation

Reports

(Usually done remotely)

ConsensusReport

(May be done remotely)

Panel report

Evaluation Summary Report

Panel ranked list

Eligibility check

Allocation of proposals to evaluators

Final ranked list

Source: European Commission

Information on evaluation outcome: 5 monthsSignature of grant agreement: 8 monthsfrom final date for submission

• Admissibility check:– General conditions on admissibility set out in General Annex B/specific

conditions in Work Programme. Proposals must be:

• submitted in the electronic submission system before the deadline given in the call

• readable, accessible and printable

– Incomplete proposals may be considered inadmissible

• Eligibility check:– General eligibility criteria set out in General Annexes A and C (e.g.

requisite minimum number of legal entities from different Member States or Associated Countries)

– Proposal must correspond to topic description

Receipt of proposals

• Experts read the proposal and evaluate against the evaluation criteria:– without discussing with others

– as submitted, not on potential if certain changes were to be made

• Excess pages marked with a watermark (experts instructed to disregard).

• Check degree to which proposal is relevant to call/topic

• Experts complete Individual Evaluation Report (IER)– Give view on operational capacity

– Give comments and scores for all evaluation criteria

– Do not recommend substantial modifications

Individual evaluation

• Selection criteria:– Financial capacity: in line with financial regulation and rules for

participation

– Operational capacity: assessed ability to carry out the project effectively

• Three Award criteria: Excellence, Impact, Quality and Efficiency of Implementation

• Each criterion scored out of 5. Threshold for each is 3, overall threshold 10 (but note specific conditions in topic descriptions).

• Innovation actions and SME instrument: impact score weighted 1.5

Evaluation

Evaluation process - Consensus

• Panel will review all the proposals within a call, or part of a call to:– ensure consensus groups have been consistent in their evaluations

– if necessary, propose new set of marks or comments

– resolve cases where a minority view was recorded in consensus report

• Panel report includes ‘Evaluation Summary Report’ (ESR) for each proposal and ‘Panel ranked list’

• If necessary, priority order for proposals with same score– Highest priority to proposals addressing topics not otherwise covered by

more highly-ranked proposals

– Highest excellence score*; then highest impact score*; then size of budget for SMEs; then gender balance in project team

Panel review

*For Innovation actions, this order is reversed

• Projects funded according to ranking within budget

• Outcome communicated to proposal coordinators:– If successful → evaluation information letter

– Not successful → proposal rejection letter

– Reserve list

• Proposal coordinators will receive and Evaluation Summary Report (ESR) showing the results of the evaluation

Finalisation

• Most calls will follow a one-stage procedure

• At first stage, only ‘excellence’ and ‘impact’ are evaluated (selected aspects only). Individual thresholds for each criterion set at 4. But unless otherwise specified, overall threshold at stage 1 will be set to achieve 1:3 success rate (budget-wise) at stage 2.

• Common feedback to coordinators of proposals that successfully pass stage 1, but first stage ESR only sent after stage two evaluation

Specific provisions relating to two-stage

proposals

New for 2016-17

Horizon 2020:

EvaluationHow to become an evaluator

• Commission maintains a database of independent experts

• Experts called upon for the:– evaluation of proposals

– review of projects

– monitoring of programmes or policies

• Need high level expertise in research or innovation in any scientific and technological field, including managerial aspects and industry expertise

• Have at least a university degree

• Have to be available for occasional, short-term assignments

• Daily fee of approximately €450 (see model contract for details)

• List of Horizon 2020 expert evaluators available

Becoming an evaluator

• In Horizon 2020 (as of 25/8/2016):– 16 825 evaluators involved in a total of 591 927 evaluations

– 66% of the evaluators came from EU-15 countries and 15% came from EU-13 countries. 6% came from Third and Associated Countries

Evaluators

Expert evaluator background

HES REC PRC OTH PUB N/A

Gender balance amongst expert evaluators

Female Male N/A

Source European Commission Horizon 2020 Monitoring Report 2015

Horizon 2020 Evaluation

criteriaExcellence, impact and implementation

• Clarity and pertinence of the objectives;

• Soundness of the concept, and credibility of the proposed methodology;

• Extent that the proposed work is beyond the state of the art, and demonstrates innovation potential (e.g. ground-breaking objectives, novel concepts and approaches, new products, services or business and organisational models)

• Appropriate consideration of interdisciplinary approaches and, where relevant, use of stakeholder knowledge

2016-2017 Award criteria: Excellence

(RIA/IA)

(bold considered for first stage of two stage applications)

• How well does the proposal fall within the topic scope?

• How ‘frontier’ and/or ‘cutting edge’ is the research?

• How ‘innovative’ in the research?

• Will it make a significant contribution and progress existing knowledge?

• Are (new) disciplinary boundaries being crossed (where appropriate)?

Excellence

• The extent to which the outputs of the project would contribute to each of the expected impacts mentioned in the work programme under the relevant topic;

• Any substantial impacts not mentioned in the work programme, that would enhance innovation capacity, create new market opportunities, strengthen competitiveness and growth of companies, address issues related to climate change or the environment, or bring other important benefits for society.

• Quality of the proposed measures to:– Exploit and disseminate the project results (including management of

IPR), and to manage research data where relevant.

– Communicate the project activities to different target audiences

2016-2017 Award criteria: Impact (RIA/IA)

(bold considered for first stage of two stage applications)

• Meeting the Expected impacts outlined in the call topics

• Think about impact in the broadest sense:– Academic impact? Publications, conferences, data management…

– Socio-economic impact? Growth, job creation, market size, IP, monitoring of exploitation potential, policy outputs, social benefits…

– Public engagement? Communication strategy, education, media, social media, user groups…

• The extent to which project outputs will contribute at European and/or international level policy, strategies, etc.– Effectiveness of the proposed project to exploit and disseminate results

to communicate the project, and to manage research data where relevant. Open Access!

Thinking about Impact

• The Impact sections has gained more importance in Horizon 2020, as the Commission now wants to see tangible outcomes of the projects in the form on new innovations (products, services, solutions, policy recommendations, etc.)

• Furthermore, there is more focus on dissemination of results and communication of the project activities.

• Impact was the most problematic section in the first applications and many applications failed because of impact not being addressed properly.

Impact

• Most applicants state that they will reach the expected impact and be involved in a number of dissemination and exploitation activities.

• No specific details of these activities are provided and that is where the applications loose points!

• The project (incl. activities to maximise impact) must appear ready to go… at least on paper!

How to address Impact correctly?

• Quality and effectiveness of the work plan, including extent to which the resources assigned to work packages are in line with their objectives and deliverables;

• Appropriateness of the management structures and procedures, including risk and innovation management;

• Complementarity of the participants and extent to which the consortium as whole brings together the necessary expertise;

• Appropriateness of the allocation of tasks, ensuring that all participants have a valid role and adequate resources in the project to fulfil that role.

2016-2017 Award criteria: Quality and

Efficiency of the Implementation (RIA/IA)

• One of the biggest mistakes made by applicants is simply describing the consortium without showing how the various partners complement each other.

• This became even more important in the 2016-17 WP with the refined wording of the Implementation criterion:

– Complementarity of the participants and extent to which the consortium as whole brings together the necessary expertise;

– Appropriateness of the allocation of tasks, ensuring that all participants have a valid role and adequate resources in the project to fulfil that role.

Implementation

• Shorter time to grant (8 months)– affects start date of project

– proposals evaluated ‘as is’: budget should be realistic to avoid reduced implementation score

• Consortium composition:– Match with activities in the proposal

– Appropriate balance of sectors

– Commission may stipulate or offer advice on types of partner/country involvement in call text

• Previous experience managing large awards, particularly EU

• Advisory Board? Decision-making processes?

Thinking About Implementation

Other considerations

• The shortage of evaluators and massive oversubscription meant that on average evaluators had approximately 2-3 hours to evaluate proposals!

• Clear and coherent structure is necessary for the evaluators to find the most important information quickly, so follow the structure provided in the templates.

• Use charts, graphs, pictures and other visuals to help evaluators review the application properly even after the first reading – HELP your evaluator give you good score

Overall presentation of the application

• Don’t underestimate gender aspects (gender experts in all Evaluation Panels)

• Relate to EU policies on Gender Equality – cross-cutting priority in Horizon 2020

• Equal opportunities (among seconded staff and decision-makers/supervisors)

• Gender dimension in the research content (e.g. subjects or end-users)

• Gender dimension in project management and networking activities

Gender Aspects

Three gender objectives in Horizon 2020:

• Fostering gender balance in Horizon 2020 research teams

• Ensuring gender balance in decision-making

• Integrating gender/sex analysis in research and innovation content

• Gender included at application stage, implementation stage and project monitoring… Written into the Grant Agreement

Gender in Horizon 2020

Always consider SSH in H2020 projects; a

must in ‘SSH-flagged topics’

• Ethics is a consideration for all EU funded projects in all research domains

• Ethics are integral to all research, from beginning to end

• Considering ethics:

– Ensures it is within the legal framework

– Enhances the quality of research

• Strong connection between research ethics and human rights

• Ethics process for Horizon 2020 – Ethics Appraisal Procedure

Ethics in Horizon 2020

Proposals to be ‘Ethics Ready’!

• Horizon 2020 mandates open access to all scientific publications.

• There are two main options:

Open Access

Green

• Self-archiving / 'green' open access – the author, or a representative, archives the published article/final peer-reviewed manuscript in an online repository before, at the same time as, or after publication.

Gold• Open access publishing / 'gold' open access - an article

is immediately published in open access mode.

• From 2017 research data is open by default in Horizon 2020 with possibilities to opt out.– This is an expansion of the original pilot across all thematic areas of

Horizon 2020

– The Commission’s moto is ‘as open as possible as closed as necessary’

• It is possible to opt out but need a justifiable reason.

Open Research Data

Source: European Commission

• Article 29.3 of the Model Grant Agreement governs the pilot and it has been made the default going forward.

• Open research data is mentioned in the introduction to the 2016-2017 work programme.

• A new General Annex L has been added to the work programme on conditions relating to open data.

• The Commission has also updated the guidance on open research data and the emphasis on FAIR data principles has been strengthened.

Open Research Data now the default

Source: European Commission

MSCA Evaluation

process

• Proposals are allocated to one of 8 evaluation panels:

Evaluation procedure

Evaluation panels

• Chemistry (CHE)

• Social Sciences and Humanities (SOC)

• Economic Sciences (ECO)

• Information Science and Engineering (ENG)

• Environment and Geosciences (ENV)

• Life Sciences (LIF)

• Mathematics (MAT)

• Physics (PHY)

Additional multidisciplinary panels for ITNs

• European Industrial Doctorates (EID)

• European Joint Doctorates (EJD)

Additional multidisciplinary panel for IFs

• Career Re-start Panel

• Reintegration Panel

• Enterprise and Society Panel

• Evaluation scores awarded for each criteria from 0 to 5

• Each award criterion has a weighting

• Total score is subject to a threshold of 70%

Award Criteria

Excellence(50%)

Impact(30%)

Implementation(20%)

Evaluation Process

• Via Participant Portal

• Admissibility/eligibility checks1. Proposal Submission

• At least 3 evaluators (often 4)

• Individual reports produced

• Each evaluator assesses ~10 proposals for ITN, ~24 proposals for IF

2. Remote Evaluations

• Consensus reports produced

• Agreement on comments/score

• Now mostly done remotely

3. Consensus Meetings

• Lists by panel

• Projects funded in priority order until budget is exhausted4. Ranked list of proposals

Max. 5 Months to Outcome!

ERC Evaluation Process

• Excellence sole evaluation criterion

• Applied to:– the ground-breaking nature, ambition and feasibility of the research

project

– the intellectual capacity, creativity and commitment of the Principal Investigator

• Proposals marked on the above, ranging from 1 (non-competitive) to 4 (outstanding)

• Numerical marks not communicated to applicants - outcome of panel meetings expressed as A, B or C (see later).

ERC evaluation criteria

• 3 research domains, 25 panels - 2 separate sets of panel members

• Indicative budget will be allocated to each panel in proportion to the budgetary demand of its assigned proposals

• Information for Applicants document provides list of panels and keywords, indicating fields of research covered

• Lists of panel members for previous ERC calls can be found on the ERC website: http://erc.europa.eu/evaluation-panels

Peer review

Social Sciences and Humanities

Physical Sciences and Engineering

Life Sciences

• SH1: Individuals, Markets and Organisations

• SH2: Institutions, Values, Environment and Space

• SH3: The Social World, Diversity, Population

• SH4: The Human Mind and Its Complexity

• SH5: Cultures and Cultural Production

• SH6: The Study of the Human Past

• PE1: Mathematics• PE2: Fundamental

Constituents of Matter• PE3: Condensed Matter

Physics• PE4: Physical and Analytical

Chemical Sciences• PE5: Synthetic Chemistry and

Materials• PE6: Computer Science and

Informatics• PE7: Systems and

Communication Engineering• PE8: Products and Processes

Engineering• PE9: Universe Sciences• PE10: Earth System Science

• LS1: Molecular and Structural Biology and Biochemistry

• LS2: Genetics, Genomics, Bioinformatics and Systems Biology

• LS3: Cellular and Developmental Biology

• LS4: Physiology, Pathophysiology and Endocrinology

• LS5: Neurosciences and Neural Disorders

• LS6: Immunity and Infection• LS7: Diagnostics, Therapies,

Applied Medical Technology and Public Health

• LS8: Evolutionary, Population and Environmental Biology

• LS9: Applied Life Sciences and Non-Medical Biotechnology

ERC panel structure

Proposal Evaluation Process

Independent, remote reviews

by panel members

(of part B1 only)

Panel meetings and ranking

Proposals retained

for stage 2, or rejected

STEP 2 - Evaluation

Interviews of PIs (StG & CoGonly), panel meetings and

ranking

Proposals selected

Independent, remote reviews by panel members and other referees of full

proposal (parts B1 and B2)

STEP 1 - Evaluation

Eligibility check

• Step 1 (Part B1 of proposal)– A: is of sufficient quality to pass to Step 2 of the evaluation

– B: is of high quality but not sufficient to pass to Step 2 of the evaluation

– C: is not of sufficient quality to pass to Step 2 of the evaluation

Applicants scoring B or C told the ranking range of their proposal out of those evaluated by the panel

• Step 2 (full proposal and interview for StG and CoG)– A: fully meets the ERC's excellence criterion and is recommended for

funding if sufficient funds are available

– B: meets some but not all elements of the ERC's excellence criterion and will not be funded

Applicants told the ranking range of their proposal out of the proposals evaluated by the panel

Outcome of evaluation

Proportions Per Score (From 2015 Calls)

Evaluated step 2 StG2015 CoG2015 AdG2015

All UK All UK All UK

A funded 46.6% 40.7% 43.7% 44.0% 42.1% 44.0%

A not funded 16.8% 10.7% 20.2% 20.0% 13.1% 14.0%

B 36.6% 48.6% 36.1% 36.0% 44.8% 42.0%

Evaluated step 1 StG2015 CoG2015 AdG2015

All UK All UK All UKA (through to step 2) 26.2% 30.7% 34.5% 41.9% 34.1% 42.3%

B 44.2% 49.5% 34.6% 36.3% 36.7% 40.7%

C 29.6% 19.8% 30.9% 21.8% 29.2% 17.0%

Further Information

• Horizon 2020 Grants Manual:http://ec.europa.eu/research/participants/portal/desktop/en/funding/reference_docs.html#h2020-grants-manual– Section on proposal submission and evaluation

– Guidance on the evaluation of some H2020 aspects (e.g. innovation, social sciences and humanities)

• Horizon 2020 Online Manual: Section on evaluation of proposals:http://ec.europa.eu/research/participants/docs/h2020-funding-guide/grants/from-evaluation-to-grant-signature/evaluation-of-proposals_en.htm

• Work Programme 2016-2017: General Annexes:http://ec.europa.eu/research/participants/data/ref/h2020/other/wp/2016-2017/annexes/h2020-wp1617-annex-ga_en.pdf

Useful links

Commission briefings for evaluators

http://ec.europa.eu/research/participants/data/support/expert/h2020_expert-briefing_en.pdf

http://ec.europa.eu/research/participants/data/ref/h2020/grants_manual/pse/h2020-evaluation-faq_en.pdf

• Gendered Innovation, Stanford University project: https://genderedinnovations.stanford.edu/

- practical tools for researchers: methods to be used in a research project; case studies; checklist

• Horizon 2020 Manual, part on Gender equality: http://ec.europa.eu/research/participants/docs/h2020-funding-guide/cross-cutting-issues/gender_en.htm

Gender Aspects - Links

• H2020 Gender Advisory Group paper on preparing grants that integrate the gender dimension into research. http://ec.europa.eu/transparency/regexpert/index.cfm?do=groupDetail.groupDetailDoc&id=18892&no=1

• https://ec.europa.eu/programmes/horizon2020/en/h2020-section/responsible-research-innovation

• Rome Declaration on Responsible Research and Innovation in Europe, November 2014https://ec.europa.eu/research/swafs/pdf/rome_declaration_RRI_final_21_November.pdf

• Report from the Expert Group on Policy Indicators for Responsible Research and Innovationhttp://ec.europa.eu/research/swafs/pdf/pub_rri/rri_indicators_final_version.pdf

• Open Science: https://ec.europa.eu/digital-agenda/en/open-science

Responsible Research and Innovation