evaluation in the h2020 calls -...
TRANSCRIPT
Evaluation in the H2020 calls
Philippe MartinetINRIA Sophia Antipolis
CHORALE TeamJune 11th 2019
Participation as reviewer for • ICT‐10‐2019‐2020 "Robotics Core Technology" (June 3rd ‐7th 2019 Brussels) (3 projects)
• SFS‐05‐2017, H2020‐2016‐2017: Robotics Advances for Precision Farming (5 projects)
• H2020‐ICT (8 projects in 2014)
• ECHORD (9 projects) and ECHORD++ (9 projects in 2014)
• STW Danemark (2015) (1 project of projects)
• National Council for Science, Higher Education and Technological Development in Croatia (2015)(Evaluation of one Excellence Center)
• FP7‐ICT (12‐13 : in average 7 per year)
Reviewer work
• Briefing meeting (visio conference)• Remote evaluation
– Reading proposals– Writing IER (Individual Evaluation Report)
• Online evaluation (Brussels or Luxembourg)– General Briefing – Specific call briefing– Consensus Meeting (1 recorder + 3 reviewers + 1 P.O ) CR– Cross reading– Panel meeting (Discussion+Calibration+debriefing)
Scoring
Results ICT‐10‐2019‐2020
• Budget 42M€• Near 90 proposals• Near 6 funded projects (6,7%)
Winners in 2019 (Accepted projects)
Exellence Impact Implentation Total Grant
5 5 4,5 14,5 8,8
5 5 4,5 14,5 5,1
4,5 5 5 14,5 6,7
5 5 4 14 7,5
5 5 4 14 7,2
4,5 4,5 4,5 13,5 6,7
Expert web briefing on 29 April 2019
Call coordinators:
Desislava Kabadzhova, H2020-DT-2019-1
Thomas R. Haub, H2020-ICT-2019-2
For your questions: CNECT-evaluators-briefing @ec.europa.eu
HORIZON 2020 PROPOSAL EVALUATION
H2020-DT-2019-1 H2020-ICT-2019-2
HORIZON 2020
2
Availability of the recorded briefing and slide deck
• Slide deck and URL of the recorded briefing will be available in the SEP evaluation tool
• namely at the bottom of your DASHBOARD under Guidance/Supporting documents
HORIZON 2020
3
Agenda
• General briefing to Evaluators 10:00
• Specific Topic Briefings 11:30 or later, as announced by your topic coordinator
HORIZON 2020
4
Content • Horizon 2020: an EU R&I programme
− More emphasis on challenge-based calls with impact
− Implementing Focus Areas or 'virtual calls'
− Cross-cutting issues
• Role of independent experts
− Confidentiality
− Conflicts of interest
• The evaluation procedure in practice
− Individual evaluation, including evaluation criteria and proposal scoring
− Consensus
− Panel review and ranking, including proposals with identical total scores
• Using the evaluation system for remote individual evaluation
HORIZON 2020 PROPOSAL EVALUATION
HORIZON 2020
6
Calls and proposals
• Calls are challenge-based, and open to innovative proposals − Calls are rarely prescriptive – they do not outline the expected
solutions to the problem, nor the approach to be taken to solve it
− Calls/topics descriptions allow plenty of scope for applicants to propose innovative solutions of their own choice
• Great emphasis on impact through 'Expected impact' statements in the Work Programme − Applicants are asked to explain how their work will bring about
described impacts
− During the evaluation, you are asked to assess this potential contribution
HORIZON 2020
7
Implementing Focus Areas through 'virtual calls' in WP 2018-20
• In Work Programme 2018-2020, there are 4 focus areas which are aligned with major political or policy drivers & expected to generate exceptional impact.
• Focus areas are 'virtually linked calls', i.e. focus areas establish the links among calls/topics from different parts of Horizon 2020.
• The rationale and objectives of the four focus areas are described in the General Introduction to the Work Programme.
• Contributing calls and topics are clearly marked (common call identifier) – in our case by the letters ‘DT’.
• The present DT call belongs to the “Digitising and transforming European industry and services” focus area. Your topic coordinator will brief you in his/her topic briefing about this Area’s content. Please take it into consideration during the evaluation.
HORIZON 2020
8
Innovation
• Balanced approach to research and innovation
− not limited to the development of new products and services on the basis of scientific and technological breakthroughs
− Incorporate use of existing technologies in novel applications, and continuous improvements
• Activities close to market emphasise the widest possible use of knowledge generated by supported activities, up to commercial exploitation
• Emphasis on activities operating close to end-users and the market, e.g. demonstration, piloting or proof-of-concept
− include support to social innovation, and support to demand side approaches (standardisation, innovation procurement, user-centred measures …) to help accelerate the deployment and diffusion of innovative products and services into the market
The definitions of the terms used are available in the Horizon 2020 Glossary on the Funding & Tenders Portal
HORIZON 2020
9
Cross-cutting issues
Cross-cutting issues integrated in the work programme (WP)
• Social Sciences and Humanities (SSH) integrated across all Horizon 2020 activities
• Gender dimension in the content of R&I - question on the relevance of sex/gender analysis is included in proposal templates. For all kinds of gender-related questions under H2020 (not just content-related ones), please see: http://ec.europa.eu/research/participants/docs/h2020-funding-guide/cross-cutting-issues/gender_en.htm
• International cooperation: The strategic approach consists of a general opening of the WP and targeted activities across all relevant Horizon 2020 parts − 'Automatic funding' to third country participants is restricted – for
information on countries eligible for funding see list of countries
• Other cross-cutting issues may also be included in the WP Responsible Research and Innovation (RRI) including science education, open access to scientific publications, ethics…; standardisation; climate change and sustainable development …
HORIZON 2020
10
Implicit gender biases may exist
• Implicit (or unconscious) gender biases refer to a cognitive phenomenon that takes place automatically and without our knowledge when assessing people and situations, influenced by our background and socio-cultural environment.
• Implicit gender biases based on gender stereotypes can affect both men and women, and influence behaviour and decision making.
• The following videos can help you understand and mitigate implicit gender biases in the evaluation process:
• Royal Society (UK) – Understanding unconscious bias
• Université de Lausanne (Switzerland) – Eviter les biais de genre lors de nominations professorales (video, with English subtitles)
• Institució CERCA (Spain) – Recruitment Bias in Research Institutes
HORIZON 2020
11
Impact of grant preparation on evaluation
No grant negotiation phase!
The time from submission of a proposal, evaluation and signature of the grant set to a maximum of 8 months
What does this mean for the evaluation of proposals?
− Evaluate each proposal as submitted not on its potential if certain changes were to be made
− If you identify shortcomings (other than minor ones and obvious clerical errors), please reflect them in a lower score for the relevant criterion
− Identify the shortcomings, but do not make recommendations i.e. do not suggest additional partners, additional work packages, resource cuts …
− Proposals with significant weaknesses that prevent the project from achieving its objectives or with resources being seriously over-estimated must not receive above-threshold scores
− Any proposal with scores above the thresholds and for which sufficient budget is available will be funded as submitted. Successful applicants are invited to address shortcomings
HORIZON 2020 PROPOSAL EVALUATION
ROLE OF INDEPENDENT EXPERTS
HORIZON 2020
13
Role of independent experts
• As an independent expert, you evaluate proposals submitted in response to a given call
• You are obliged to assess the proposals yourself – you must not delegate this to anybody else
• You must carry out this task in the evaluation IT tool
• Significant funding decisions will be made on the basis of your assessment
• You need not comment on ethics issues anymore, as successful proposals will undergo an ethics review
• However, if you suspect any form of research misconduct (e.g. plagiarism, falsification, double funding), please notify Commission staff
HORIZON 2020
14
Guiding principles • Independence
− You are evaluating in a personal capacity − You represent neither your employer, nor your country!
• Impartiality
− You must treat all proposals equally and evaluate them impartially on their merits, irrespective of their origin or the identity of the applicants
• Objectivity
− You evaluate each proposal as submitted; i.e. on its own merit, not on its potential if certain changes were to be made
• Accuracy
− You make your judgment against the official evaluation criteria and the call/topic the proposal addresses, and nothing else
• Consistency
− You apply the same standard of judgment to all proposals
HORIZON 2020
15
Confidentiality You must:
• Not discuss evaluation matters, such as the content of proposals, the opinions of fellow experts or evaluation results, with anyone, including: − Other experts, Commission staff or any other person (e.g. colleagues, students…)
not directly involved in the evaluation of the proposal − The sole exception: your fellow experts who are evaluating the same proposal in a
consensus group or Panel review and the assigned Commission staff
• Not contact partners in the consortium, sub-contractors or any third parties
• Never disclose the names of your fellow experts
• Never disclose your own name related to a specific call, topic or proposal − The Commission publishes the names of the experts annually - as a group, no link
can be made between an expert and a proposal
• Maintain confidentiality of documents, paper or electronic, at all
times and wherever you do your evaluation work
− Return, destroy or delete all confidential documents, paper or electronic, upon completing your work
HORIZON 2020
16
Conflicts of interest (CoI) (1)
You have a CoI, if you: • were involved in the preparation of the proposal (including
pre-proposal checks / 'mock' evaluations)
• If you are involved in a competing proposal
• stand to benefit directly/indirectly, if the proposal is successful or fails
• have a close family/personal relationship with any person representing an applicant legal entity
• are a director/trustee/partner of an applicant or involved in the management of an applicant's organisation
• are employed or contracted by an applicant or a named subcontractor
• are a member of an Advisory Group or Programme Committee
• are a National Contact Point or are directly working for the Enterprise Europe Network
HORIZON 2020
17
Conflicts of interest (CoI) (2)
You may have a CoI, if you:
− Were employed by an applicant (including third parties or linked third parties involved in the proposal) in the last 3 years
− Were involved in the last 3 years:
− in a grant agreement with an applicant / a grant decision of or for an applicant;
− in the management structure of an applicant;
− in a research collaboration with an applicant.
− Are in any other situation that casts doubt on your impartiality or that could reasonably appear to do so
If you are in any of these situations, the Com-mission will decide whether or not there is a COI
COI rules are spelled out in Annex 1 'Code of Conduct' of your expert contract.
HORIZON 2020
18
Conflicts of interest (CoI) (3)
• You must inform the Commission as soon as you become aware of a COI
− Before the signature of the contract
− Upon receipt of proposals, or
− During the course of your work
• If there is a COI for a certain proposal you cannot evaluate it
− Neither individually
− Nor in the consensus group
− Nor in the panel review
− For the 'soft' CoIs, the Commission will determine if there is a COI on a case-by-case basis and decide on the course of action to follow
• If you knowingly hide a COI, you will be excluded from the evaluation and your work declared null and void
− The allowance/expenses you claimed may be reduced, rejected or recovered
− Your contract may be terminated
HORIZON 2020 PROPOSAL EVALUATION
THE EVALUATION PROCEDURE IN PRACTICE
HORIZON 2020
20
Overview of the Evaluation Process
Receipt of proposals
Individual evaluation
Consensus group
Panel Review Finalisation
Evaluators
Individual Evaluation Reports
(Usually
done remotely)
Consensus Report
(May be done
remotely)
Panel report
Evaluation Summary Report
Panel ranked list
Eligibility check
Allocation of proposals to evaluators
Final ranked list
HORIZON 2020
21
Admissibility, eligibility checks and additional requirements
• Admissibility is checked by the Commission:
− Readable, accessible and printable
− Completeness of proposal, presence of all requested forms
− Inclusion of a plan for exploitation and dissemination of results
• Eligibility checked by the Commission - however, if you spot an issue relating to eligibility, please inform Commission staff
− Minimum number of independent partners as set out in the call conditions
• “Out of scope” – content of a proposal corresponds, wholly or in part, to the description of the call or topic
− A proposal will only be deemed ineligible in clear-cut cases, where there is no obvious link between proposal and call topic
HORIZON 2020
22
Countries whose entities are eligible for funding
Member States of the European Union, including their overseas departments and outermost regions.
Associated Countries – Iceland, Norway, Albania, Bosnia and Herzegovina, North Macedonia, Montenegro, Serbia, Turkey, Israel, Moldova, Switzerland, Faroe Islands, Ukraine, Tunisia, Georgia, Armenia
Third Countries – see the 'Annex A - List of countries, and applicable rules for funding' for the list of third countries that are eligible for funding.
Exceptionally, other countries if:
• Bilateral agreement, e.g. EU-US/NIH arrangement
• Identified in the Work Programme
• Deemed essential for carrying out the action • Outstanding competence / expertise
• Access to research infrastructure
• Access to data
• Access to particular geographical environments
HORIZON 2020
23
Outcome of the UK referendum and Horizon 2020: State of Play
Until the UK leaves the EU, EU law continues to apply to and within the UK, both when it comes to rights and obligations. This includes the eligibility of UK legal entities to participate in and receive funding from Horizon 2020 actions. Experts should therefore not evaluate proposals with UK participants any differently than before.
HORIZON 2020
24
Evaluation criteria
• There are three evaluation criteria:
1. Excellence (in relation to the challenge posed by the call/topic)
2. Impact
The 'Impact' criterion mostly refers to the customised 'Expected Impact' statements, which are part of any topic of the Work Programme. You must evaluate the proposals against these statements, when assessing 'Impact'.
3. Quality and efficiency of the implementation
You should also check requests for ‘exceptional funding’ from third country participants not included in the list
If your topic allows financial support to 3rd parties, look into the proposal's related activities and check that this support will be given in compliance with Annex K to the WP
You will also be asked if the proposal foresees the use of human embryonic stem cells (hESC) – this is irrelevant for almost all ICT-related proposals
• Evaluation criteria are adapted to each type of action, as specified in the WP (please see slides further down for details)
HORIZON 2020
25
Open Research Data Pilot & Research data management plan
• Participation in the extended Open Research Data Pilot (ORDP) is now the default option for all proposals. However, consortia are allowed to opt out of the pilot.
• Participation in the pilot is NOT subject to evaluation! So please do not penalise proposals that have chosen to opt out of the ORDP.
• Proposals participating in the Open Research Data Pilot (ORDP) need a Research Data Management Plan. Yet again, this plan is not subject to evaluation!
• Further guidance is available in the H2020 Online Manual on both open access to research data and research data management.
HORIZON 2020
26
Type of actions
Research and Innovation Action
• Action primarily consisting of activities to establish new knowledge and/or explore feasibility of new or improved technology, product, process, service or solution
− May include basic and applied research, technology development and integration, testing and validation on small-scale prototype in laboratory or simulated environment
− Projects may contain closely connected but limited demonstration or pilot activities to show technical feasibility in a near to operational environment
− Page limit for sections 1-3 of the proposal's Part B: 70 pages
HORIZON 2020
27
Type of actions
Innovation Action
• Action primarily consisting of activities that aim to produce plans, arrangements or designs for new, altered or improved products, processes or services
− May include prototyping, testing, demonstrating, piloting, large-scale product validation and market replication
− Aim to validate the technical and economic viability in a (near) operational environment and / or support the first application / deployment in the market of an innovation that has already been demonstrated but not yet applied / deployed in the market due to market failures / barriers to uptake
− Projects may include limited research and development activities
− Page limit for sections 1-3 of the proposal's Part B: 70 pages
HORIZON 2020
28
Type of actions
Coordination & Support Action (CSA)
• Actions consisting primarily of accompanying measures such as
− standardisation, dissemination, awareness-raising and communication, networking, coordination or support services, policy dialogues and mutual learning exercises and studies, including design studies for new infrastructure, and
− may also include complementary activities of strategic planning, networking and coordination between programmes in different countries
− Page limit for sections 1-3 of the proposal's Part B: 50 pages
HORIZON 2020
29
Type of actions
Pre-Commercial Procurement (PCP) Action • Encourage public procurement of research, development and
validation of new solutions that can bring significant quality and efficiency improvements in areas of public interest, while opening market opportunities for industry and researchers
− Provides EU co-funding for group of procurers to undertake together one joint PCP procurement, so that there is one joint call for tender, one joint evaluation of offers, and a lead procurer awarding the R&D service contracts in the name and on behalf of the group
− Each procurer contributes its individual financial contribution to the total budget necessary to jointly finance the PCP, enabling the procurers to share the costs of procuring R&D services from a number of providers and comparing together the merits of alternative solutions paths to address the common challenge
− The PCP shall explore alternative solution paths from a number of competing providers to address one concrete procurement need that is identified as a common challenge in the innovation plans of the procurers that requires new R&D
− Cross-border PCP cooperation should better address issues of common European interest, for example where interoperability and coherence of solutions across borders is required
• Page limit for sections 1-3 of the proposal's Part B: 70 Pages
HORIZON 2020
30
The following aspects will be taken into account, to the extent that proposed work corresponds to the topic description in the work programme: • Clarity and pertinence of the objectives • Soundness of the concept, and credibility of the proposed methodology • Extent that proposed work is beyond the state of the art, and demonstrates innovation potential (e.g.
ground-breaking objectives, novel concepts and approaches, new products, services or business and organisational models)
• Appropriate consideration of interdisciplinary approaches and, where relevant, use of stakeholder knowledge and gender dimension in research and innovation content.
Excell
en
ce
The following aspects will be taken into account: • The expected impacts listed in the work programme under the relevant topic • Any substantial impacts not mentioned in the WP, that would enhance innovation capacity; create new
market opportunities, strengthen competitiveness and growth of companies, address issues related to climate change or the environment, or bring other important benefits for society
• Quality of proposed measures to exploit and disseminate project results (including IPR, manage research data where relevant); communicate the project activities to different target audiences
Im
pact
Research and Innovation Actions/Innovation Actions
The following aspects will be taken into account: • Quality and effectiveness of the work plan, including extent to which resources assigned in work
packages are in line with objectives/deliverables • Appropriateness of management structures and procedures, including risk and innovation management • Complementarity of the participants and the extent to which the consortium as a whole brings together
the necessary expertise • Appropriateness of allocation of tasks, ensuring that all participants have a valid role and adequate
resources in the project to fulfill that role Im
ple
men
tati
on
Evaluation criteria
HORIZON 2020
31
Coordination & Support Actions
The following aspects will be taken into account, to the extent that proposed work corresponds to the topic description in the work programme: • Clarity and pertinence of the objectives • Soundness of the concept, and credibility of the proposed methodology • Quality of the proposed coordination and/or support measures
Excell
en
ce
The following aspects will be taken into account:
• The expected impacts listed in the work programme under the relevant topic • Quality of proposed measures to:
- Exploit and disseminate project results (including IPR, manage research data where relevant); - Communicate the project activities to different target audiences
Im
pact
The following aspects will be taken into account: • Quality and effectiveness of the work plan, including extent to which resources assigned in work
packages are in line with objectives/deliverables • Appropriateness of management structures and procedures, including risk and innovation management • Complementarity of the participants and the extent to which the consortium as a whole brings together
the necessary expertise • Appropriateness of allocation of tasks, ensuring that all participants have a valid role and adequate
resources in the project to fulfill that role
Im
ple
men
tati
on
Evaluation criteria
HORIZON 2020
32
The following aspects will be taken into account, to the extent that proposed work corresponds to the topic description in the work programme: • Clarity and pertinence of the objectives • Soundness of the concept, and credibility of the proposed methodology • Progress beyond the state of the art in terms of the degree of innovation needed to satisfy the
procurement need
Excell
en
ce
The following aspects will be taken into account: • The expected impacts listed in the work programme under the relevant topic • Strengthening the competitiveness and growth of companies by developing innovations meeting needs
of European and global procurement markets • Quality of proposed measures to exploit and disseminate project results (including IPR, manage
research data where relevant); communicate the project activities to different target audiences • More forward looking procurement approaches reducing fragmentation of demand for innovative
solutions
Im
pact
The following aspects will be taken into account: • Quality and effectiveness of the work plan, including extent to which resources assigned in work
packages are in line with objectives/deliverables • Appropriateness of management structures and procedures, including risk and innovation management • Complementarity of the participants and the extent to which the consortium as a whole brings together
the necessary expertise • Appropriateness of allocation of tasks, ensuring that all participants have a valid role and adequate
resources in the project to fulfill that role Im
ple
men
tati
on
Pre-Commercial Procurement Actions (PCP)
Evaluation criteria
HORIZON 2020
33
Operational capacity & financial support to 3rd parties
• As part of the individual evaluation, give your view on whether each applicant has the necessary basic operational capacity to carry out their proposed activity(ies) based on the information provided in section 4 of the proposal's Part B: − Curricula Vitae or description of the applicant's profile − Relevant publications or achievements − Relevant previous projects or activities − Description of any significant infrastructure or any major items of technical
equipment − Description of 3rd parties contributing to the work but not represented as project
partners In exceptional cases, please assess the concrete measures proposed to obtain operational capacity by the time of project implementation
• At the consensus group, you consider together whether an applicant lacks basic operational capacity
• If an applicant lacks basic operational capacity, the consensus group comments on the proposal and gives scores without taking into account this applicant and its activity(ies)
• If your topic allows financial support to 3rd parties, look into the proposal's related activities and check that this support will be given in compliance with Annex K to the WP
HORIZON 2020
34
Proposal scoring
• Reflecting your comments, you give a score between 0 and 5 to each
criterion
− The score summarises the set of comments under the criterion in question
− The whole range of scores should be used; use steps of 0.5
− Scores must pass thresholds if a proposal is to be considered for funding
• Thresholds apply to individual criteria …
The threshold is 3
• … and to the total score
The total-score threshold is 10
• For Innovation actions, to determine the ranking, the score for
criterion "Impact" will be given a weight of 1.5
HORIZON 2020
35
Interpretation of the scores
The proposal fails to address the criterion or cannot be assessed due to missing or incomplete information.
Poor. The criterion is inadequately addressed, or there are serious inherent weaknesses.
Fair. The proposal broadly addresses the criterion, but there are significant weaknesses.
Good. The proposal addresses the criterion well, but a number of shortcomings are present.
Very Good. The proposal addresses the criterion very well, but a small number of shortcomings are present.
Excellent. The proposal successfully addresses all relevant aspects of the criterion. Any shortcomings are minor.
0
1
2
3
4
5
HORIZON 2020
36
Evaluation Process
Individual Evaluation
Report
Individual Evaluation
Report Individual Evaluation
Report
Consensus group
Consensus Report
Individual Evaluation
Report
Individual Evaluation
Report
Expert Expert Expert Expert Expert Minimum 3 experts … but can be more
Individual evaluation
Consensus
Proposal Eligible proposal
HORIZON 2020
37
Individual evaluation
• Read the proposal and evaluate it against the evaluation criteria
− Without discussing it with anybody else
− As submitted - not on its potential if certain changes were to be made
− Do not penalise applicants that did not provide detailed, broken-down costs – they are not required
• Disallowed excess pages are blanked out and marked with a watermark
• Fill in an Individual Evaluation Report (IER)
− Give comments and scores for all evaluation criteria (scores must reflect comments)
− Identify shortcomings, but do not make recommendations (e.g. no additional partners, work packages, resource cuts)
• Check to what degree the proposal is relevant to the call or topic
• Give your view on operational capacity
• Sign and submit the form in the electronic system
Look at the substance: Some proposals may be
handicapped by language difficulties,
others deceptively well written
HORIZON 2020
38
If a proposal • Is only marginally relevant, in terms of its scientific, technological
or innovative content, to the call or topic addressed, you must reflect this in a lower score for the Excellence criterion
− No matter how excellent the science!
• Does not significantly contribute to the expected impacts as specified in the WP for that call or topic, you must reflect this in a lower score for the Impact criterion
• Would require substantial modifications in terms of implementation (i.e. change of partners, additional work packages, significant budget or resource cuts …), you must reflect this in a below-threshold score for the “Quality and efficiency of the implementation” criterion
• If cross-cutting issues are explicitly mentioned in the scope of the call or topic, and not properly addressed (or their non-relevance justified), you must reflect this in a lower score for the relevant criterion
− A successful proposal is expected to address cross-cutting issues, or convincingly explain why they are not relevant in its particular case
− Proposals addressing cross-cutting issues which are not explicitly mentioned in the scope of the call or topic can also be evaluated positively
HORIZON 2020
39
Logistics
• For the electronic evaluation system you need your "EU Login" credentials
− Please make sure you know the e-mail address and password that you use for "EU Login"
• Please bring your own device
− To the on-site evaluation in Brussels or Luxembourg, you are invited to bring along your own laptop/tablet/notebook (incl. chargers, adapters [VGA, HDMI cables], etc.)
Laptops are available upon request to make up for urgencies
Desk-top computers are installed in some meeting rooms
• Electronic workflow
− Your payment requests are also processed electronically (no more queues for reimbursement)
HORIZON 2020 PROPOSAL EVALUATION
USING THE EVALUATION SYSTEM FOR REMOTE
INDIVIDUAL EVALUATION
HORIZON 2020
41
Use your "EU Login" e-mail address and then your password to enter the system.
Once connected your list of active tasks is displayed in SEP.
Connect to the SEP evaluation system (2 options):
1. follow this link: https://ec.europa.eu/research/participants/evaluation 2. or google "SEP EU" and follow the first link that appears in the search results
Proposal reference
Acronym
How to log on
HORIZON 2020
42
Read a proposal
a) from the task list, click on the proposal reference or acronym.
This action opens the task details screen, where you have access to the proposal.
Check potential CoI and decline the task, if there is any.
b) If no CoI detected click Edit task
You may save the report at any time, and return to it later. Submit when you have completed your individual assessment. Complete and submit your reports one by one. Once submitted, the report is no longer editable, but is still accessible from the "All tasks" List using the View button. Submission by mistake can be undone.
Write an IER
The deadline for completing
IERs will be communicated to you by your
topic coordinator
Please do not wait until the last moment; submit your IERs as you finalise them!
HORIZON 2020
44
Where to find the recorded briefing and slide deck
• Slide deck and URL of the recorded briefing will be available in the SEP evaluation tool
• namely at the bottom of your DASHBOARD under Guidance/Supporting documents
Antonio Puente Topic coordinator ICT-09 & ICT-10 Robotics & AI European Commission
ROBOTICS in H2020 - ICT-2019-2
Topics ICT-09 and ICT-10
Contents
• Sources of information and background to the evaluation
• Presentation of the Work Programme topics
• ICT-09 Robotics in application areas
• ICT-10 Robotics core technology
• Frequently asked questions and answers
• Hints for drafting Individual Evaluation Reports (IERs)
Information about the call topics:
ICT-09-2019-2020: Robotics in Application Areas
http://ec.europa.eu/research/participants/portal/desktop/en/opportunities/h2020/t
opics/ict-09-2019-2020.html
ICT-10-2019-2020: Robotics Core Technology
http://ec.europa.eu/research/participants/portal/desktop/en/opportunities/h2020/t
opics/ict-10-2019-2020.html
DT-ICT-02-2018: Robotics Digital innovation Hubs
https://ec.europa.eu/info/funding-
tenders/opportunities/portal/screen/opportunities/topic-details/dt-ict-02-2018
WORK PROGRAMME
http://ec.europa.eu/research/participants/data/ref/h2020/wp/2018-
2020/main/h2020-wp1820-leit-ict_en.pdf
SRA + MAR (SPARC)
https://www.eu-robotics.net/cms/upload/topic_groups/SRA2020_SPARC.pdf
https://www.eu-robotics.net/cms/upload/topic_groups/H2020_Robotics_Multi-
Annual_Roadmap_ICT-2017B.pdf
Additional information:
4 Priority Application Areas
Healthcare
Inspection and maintenance
of infrastructure
Agri-food
Agile production
TIMELINE
2018
DT-ICT-02-2018
DIGITAL INNOVATION HUBS
17 APRIL 2018
IA 64M€ CSA 2M€
2019 2020
ICT-09-2019-2020:
APPLICATION AREAS ICT-10-2019-2020:
CORE TECHNOLOGY
28 MARCH 2019
ICT-09-2019-2020:
APPLICATION AREAS ICT-10-2019-2020:
CORE TECHNOLOGY DT-ICT-12-2020:
THE SMART HOSPITAL OF THE FUTURE
ICT09: RIA 20M€ ICT09: IA 28M€ ICT09: CSA 2M€ ICT10: RIA 42M€
825263 RODIN – CSA: coordinating the five DIH networks
824964 DIH^2 – IA: DIH network on agile production
825196 TRINITY – IA: DIH network on agile production
825196 RIMA – IA: DIH network on infrastructure inspection and maintenance
825003 DIH-HERO – IA: DIH network on healthcare
825395 AGROBOFOOD – IA: DIH network on agro-food (not started yet)
RESULT OF LAST YEAR CALL: TOPIC DT-ICT-02-2018: Digital Innovation Hubs
DIHs DT-ICT-02-
2018
COORD & SUPPORT
DT-ICT-02-2018
PILOTS
ICT-09 b) 28 M€
COMPETITIONS
ICT-09 c) 2 M€
CORE TECH
ICT-10 42 M€
4 PRIORITY APPLICATION AREAS:
INSPECTION & MAINTENANCE OF INFRASTRUCTURE
AGRI-FOOD AGILE PRODUCTION
HEALTHCARE
R&I BOOST
APPLICAT
ICT-09 a) 20 M€
Call 2018 Call 2019
TOPIC: ICT-09-2019-2020: Robotics in Application Areas
Scope of the topic:
a) Innovative approaches to hard research problems in new promising robotics applications
– RIA: 3-5M€/action – TOTAL: 20M€
b) Large scale pilots in Robotics demonstrating the use of robotics in highly realistic environments of infrastructure inspection and maintenance
- IA: ~7-9M€/action – TOTAL: 28M€
c) Robotic competitions in healthcare, inspection and maintenance of infrastructure, agri-food, and agile production
- CSA: ~2M€/action – TOTAL: 2M€
http://ec.europa.eu/research/participants/portal/desktop/en/opportunities/h2020/topics/ict-09-2019-2020.html
TOPIC: ICT-09-2019-2020: Robotics in Application Areas
Stakeholders expected in the proposals:
Academia and industry developing or using intelligent robots, and end-users. The involvement of end-users is particularly important in b) and c)
Specific Challenges (technical and non-technical):
Reduce the barriers to adoption
Satisfy user needs
Raise awareness and take-up
Ethical, legal, societal and economic aspects
Privacy and cybersecurity issues, where appropriate
TOPIC: ICT-09-2019-2020: Robotics in Application Areas
Scope a): Research and Innovation boosting promising robotics applications (Research and Innovation Action)
Innovative approaches to hard research problems in relation to applications of robotics in promising new areas
Enable substantially improved solutions to challenging technical issues, with a view of take-up in applications with high socio-economic impact.
Driven by application needs
Can start from low TRL, but validate results in realistic environments to demonstrate the potential for take-up
Open to all robotics-related research topics and to all new application areas (The four priority areas* are excluded);
The proposal can addresses multiple application areas.
Scope a): Research and Innovation boosting promising robotics applications
Connect and cooperate with the DIHs*, platforms and other relevant activities of this work programme, as appropriate
*DIHs can be others than those that have been granted with the call DT-ICT-02-2018;
for info please visit the DIH catalogue:
http://s3platform.jrc.ec.europa.eu/digital-innovation-hubs-tool
*Proposals need to allocate budget for meetings, make sure it is included in dissemination and outreach plans and within the description of an appropriate work package or task;
*A partner needs to be responsible to make the connections and act as a single point of contact between the DIH networks and the Funded Action
TOPIC: ICT-09-2019-2020: Robotics in Application Areas
TOPIC: ICT-09-2019-2020: Robotics in Application Areas
Scope a): Research and Innovation boosting promising robotics applications
Expected Impact
Strengthening European excellence in Robotics S&T
Boosting the use of robotics in promising application areas
Opening up new markets for robotics
Lowering barriers in the deployment of robotics-based solutions
Scope b): Large scale pilots in Robotics for infrastructure inspection and maintenance
Demonstrate the use of robotics at scale in actual or highly realistic operating environments;
Showcase advanced prototype applications built around platforms operating in real or near-real environments;
Demonstrate high levels of socio-economic impact;
Utilise existing infrastructure and links to other European, national or private funding-sources;
Identify the long-term sustainability of the pilot;
Develop scalable technical solutions capable of meeting performance targets; develop metrics and performance measures for the pilot;
Disseminate best practice and coordinate access to platforms and demonstrators.
TOPIC: ICT-09-2019-2020: Robotics in Application Areas
Scope b): Large scale pilots in Robotics for infrastructure inspection and maintenance
Expected Impact:
Demonstrate potential for robotics to impact at scale
Reduce technical and commercial risk in the deployment of services based on robotic actors
Greater understanding from the application stakeholders of the potential for deploying robotics
Demonstrate platforms operating over extended time periods in near realistic environments and promote of their use
Develop the eco-system to stimulate deployment
Contribute to the development of open, industry-led or de facto standards
TOPIC: ICT-09-2019-2020: Robotics in Application Areas
Scope c): Robotic competition in healthcare, inspection and maintenance of infrastructure, agri-food, and agile production
Reducing technical and commercial risks: performance data
Real/near-real operating environment for long-term trials & the testing of deployment strategies
Challenge-led, robotics competitions focusing on the four PAs
Stimulate public engagement & engage with the Robotics DIHs
All aspects of running competitions as public events, and engage with the media and public
Proposals should seek to mobilise external partners in sponsoring and setting up the competitions
TOPIC: ICT-09-2019-2020: Robotics in Application Areas
Scope c): Robotic competition in healthcare, inspection and maintenance of infrastructure, agri-food, and agile production
Expected Impact:
Greater public exposure to actual robotics capability.
Greater engagement with competitions from commercial organizations in the four priority application areas
TOPIC: ICT-09-2019-2020: Robotics in Application Areas
TOPIC: ICT-10-2019-2020: Robotics Core Technology Objectives: Increased autonomy in robotics systems
through research in:
AI and Cognition
Cognitive Mechatronics
Socially cooperative human-robot interaction
Model-based design and configuration tools
Stakeholders expected in the proposals:
Academia and industry developing or using intelligent robots
Link to the call text: http://ec.europa.eu/research/participants/portal/desktop/en/opportunities/h2020/topics/ict-10-2019-2020.html
TOPIC: ICT-10-2019-2020: Robotics Core Technology AI and Cognition:
Equips robots with the ability to interact with people and environments, to learn and to categorize, to make decisions and to derive knowledge
Cognitive Mechatronics:
Mechatronic systems where sensing and actuation are closely coupled with cognitive systems
Improved control, motion, interaction, adaptation and learning, and safer systems
TOPIC: ICT-10-2019-2020: Robotics Core Technology Socially cooperative human-robot interaction:
Cooperative human-robot interaction
Critical in many work environments: from collaborative support (e.g. passing tools to a worker), to the design of exo-skeletons able to provide motion that is sympathetic to the user
Model-based design and configuration tools:
Deploying robotics at scale in application areas where tasks need to be defined by the user requires easy-to-use configuration tools.
Embedding and sharing of knowledge between tools is essential, as is standardization across the interfaces to connect systems and modules
Taking into account cybersecurity issues
TOPIC: ICT-10-2019-2020: Robotics Core Technology
Scope:
Development of core technology modules (modular, open and non-proprietary) and tool kits for use in deployable system platforms that meet the requirements of applications in the four prioritised application areas: Healthcare, Infrastructure Inspection and Maintenance, Agri-Food and Agile Production.
Proposals should address at least one of the four areas
Dedicate resource for connecting with the DIH actions arising from DT-ICT-02-2018.
TOPIC: ICT-10-2019-2020: Robotics Core Technology
Expected impact:
Improved technical capability
Greater range of applications in the prioritised application areas that can be demonstrated at TRL 3 and above.
Lowering technical barriers within the prioritised applications areas
Frequently Asked Questions and Answers (Q&A)
• Under call H2020-ICT-09-2019-2020: are the four Priority Areas (PAs) addressed in all subtopic?
• The four PAs mentioned in the first paragraph of the description of topic ICT-09-2019-2020 are the default areas unless stated differently. Therefore the actions proposed under sub-topic:
• a) are excluded from addressing the four indicated priority areas;
• b) are expected to address the priority area of infrastructure inspection and maintenance;
• c) are expected to address the four priority areas.
•
Q&A
• Under the call H2020-ICT-09-2019-2020: Do the proposals need to show that they allow for connectivity with DIHs network?
• Yes, they need to connect with DIHs network specifically represented in the subtopics as follows:
• a) DIHs listed in the workprogramme;
• b) Robotics DIH network established in DT-ICT-02-2018;
• c) Robotics DIH network established in DT-ICT-02-2018.
• Under the call H2020-ICT-09-2019-2020 scope c): How many priority areas need to be addressed by the competitions?
• The competitions need to be developed in all the priority areas;
Q&A
• Under call H2020-ICT-10-2019-2020: Proposals are required to dedicate resource for connecting with the DIH actions arising from DT-02-2018. How can proposers access the list of accepted DIH actions from DT-ICT-02-2018?
• The list of accepted actions from DT-ICT-02-2018 is not essential for preparing proposals for the topic ICT-10-2019-2020. The information contained in the specification of the call topic DT-ICT-02-2018 (http://ec.europa.eu/research/participants/portal/desktop/en/opportunities/h2020/topics/dt-ict-02-2018.html) must be enough for planning the efforts to connect and cooperate with the DIH actions.
Q&A
• Under call H2020-ICT-10-2019-2020: what is the definition of Agile production
• Agile Production refers to robotic production systems that operate quickly and adaptively in dynamically changing work contexts. They can adapt to varying work tasks, varying work pieces and varying schedules of work, where adaptation is carried out at speed in comparison to the time scale of the task. The term “Agile” refers to speed and adaptation in combination and is related to the system’s execution of a task or the speed with which reconfiguration or adaptation to a different task can be carried out. In successful “Agile Production” there is an implication that tasks will maintain or improve efficiency, accuracy and dependability over non-agile production. “Agile Production” includes anything that is made, all manufactured goods; food, clothes, shoes, pharmaceuticals, craft items, components and assemblies, buildings, transport infrastructure etc.
•
Q&A
• Under call H2020-ICT-10-2019-2020: How many core technologies need to be addressed?
• Proposals should address one of the four core technologies.
Hints for drafting IERs
• Start by telling us if the proposal is in scope and a one-sentence description of the proposal (needed for portfolio analysis)
• Write at least one sentence under each of the sub-criteria (bullet points) under the three main criteria (but do not score the sub-criteria separately)
• Be specific and refer to concrete instances in the proposal.
• Explain why a given aspect is poor or fair or good or very good or excellent
• Draft the statements first; the score follows from the text.
• Your statements are essential for the on-site discussions – please write an informative and self-contained IER.
Frequently misunderstood issues (1)
• Management by a professional consultancy
• is not to be taken for granted
• must offer a proven added-value and cost efficient solution
• Applications and products
• note the different role of applications in RIAs and IAs
• nevertheless, we are not looking for commercial product development within the project lifecycle
• Participation in EU projects in the past
• is not necessarily to be considered an advantage
• progress beyond the state of the art including the achievements of previous projects must be clearly demonstrated
Frequently misunderstood issues (2)
• European level
• does not mean having a spread of partners from countries all over Europe without proper justification regarding the project in hand
• means cross-border collaboration that promises to achieve more than could have been achieved within one single state
• Presence of one or more SMEs
• is not to be taken as a must or as a de facto plus point
• SMEs are treated just as any other partner in a consortium in terms of having the necessary competence and a clearly defined role
Individual evaluation tasks will be allocated on SEP from tomorrow. Please check and signal CoI first. Consensus report recording will be carried out during the central evaluation in Brussels or Luxembourg
Deadline for the submission of your IERs: 28th May 2018 for the topic ICT-10 13th June 2019 for the topic ICT-09
DG CONNECT – UNIT R5 Programme Implementation
Information and Communications Technologies
Recorder briefing
Calls 2018/2020
Availability of the recorded briefing and slide deck
• Slide deck and URL of the recorded briefing will be available in the SEP evaluation tool
• namely at the bottom of your DASHBOARD under Guidance/Supporting documents
Welcome
and thank you for helping us record the evaluation
of the proposals submitted to this call!
Evaluation
You will be assigned proposals for the recording task
On the Consensus Reports (CR) related to those proposals, please record as accurately and completely as possible the conclusions reached by the evaluators
Never put down your own view on a proposal, please
Please treat any proposal information and any opinions, which you hear, as confidential, both now and in the future
Your responsibilities 1
You will never be asked to act as Recorder for a proposal that involves your employer or your own company
If you are requested to act as Recorder for a proposal, for which your impartiality could be put in doubt, please let us know immediately, and we will re-assign the proposal to another Recorder
Your responsibilities 2
Stage 1. Individual readings • Each proposal is read independently by three or more experts
and evaluated under three specific criteria:
1. Excellence 2. Impact 3. Implementation They also assess the applicants' operational capacity to carry out the work and give their advice regarding the question of whether the proposal is in the scope of the call
• Per proposal, each expert writes and submits an Individual Evaluation Report (IER) using the Evaluation Service tool
• The Recorder has access to the proposals and the IERs. As soon as the first IER has been submitted, s/he may already start drafting the Consensus Report (CR) for this proposal
Proposal evaluation 1
Stage 2. Consensus Group • The experts, who have evaluated the proposal
individually, congregate in Brussels (or Luxembourg)
• The group's objective is to come to a consensual view on the proposal. This view will then be retained in the Consensus Report (CR)
• In this task, the evaluators are supported by a Commission Moderator and you, the proposal Recorder
Proposal evaluation 2
Stage 3. Panel meeting • All the experts contracted for a certain topic constitute
the panel, which reviews the Consensus reports of all proposals
• For every proposal, the panel prepares an Evaluation Summary Report ESR, which will later be sent to the applicants
• The panel is supported by a Commission Panel
Coordinator and a Panel Recorder
Proposal evaluation 3
• Under each evaluation criterion, explanatory comments are made and a score is given between 0 and 5. The score must reflect the set of comments
• A threshold must be passed for each criterion
• An overall score is calculated for each proposal by simple addition
• In most cases, a threshold must also be passed for the overall score
• Some topics may have different thresholds and/or apply
a weighting scheme (e.g. FET). Innovation Actions (IA) always apply a weighting scheme for the ranking.
The scoring scale 1
The scoring scale 2
The proposal fails to address the criterion or cannot be assessed due to missing or incomplete information.
Poor. The criterion is inadequately addressed, or there are serious inherent weaknesses.
Fair. The proposal broadly addresses the criterion, but there are significant weaknesses.
Good. The proposal addresses the criterion well, but a number of shortcomings are present.
Very Good. The proposal addresses the criterion very well, but a small number of shortcomings are present.
Excellent. The proposal successfully addresses all relevant aspects of the criterion. Any shortcomings are minor.
0
1
2
3
4
5
Evaluation service tool
Preparing the consensus report in SEP
Use your "EU Login" e-mail address and your password to enter the system.
Once connected your active task list is displayed in SEP.
Connect to the SEP evaluation system (2 options):
1. follow this link: https://ec.europa.eu/research/participants/evaluation 2. or google "SEP EU" and follow the first link that appears in the search results
Proposal reference
Acronym
How to log on
How to read the proposal
a) from the Task List, click on either the proposal reference or acronym.
This action opens the task details screen, where you have access to the proposal.
Check potential CoI with the partners of the project, and decline the task if necessary
b) From the task details, click on Part A and check again for CoI, decline the task if necessary
c) If no CoI detected click Edit task
You may save the report at any time, and return to it later. Don't push the "submit" button, as you will again work on the draft in Brussels (or Luxembourg). Once submitted, the report is no longer editable, but is still accessible from the "All tasks" List using the View button. Submission by mistake can be undone.
Prepare the draft CR
1) Initialize: to integrate IERs into draft CR
2) Merge IER: pop-up window showing finalised IERs
Each expert writes an (IER) Individual Evaluation Report
Evaluation process
Consensus Report (CR)
Each proposal is read independently by at least 3 experts and evaluated under 3 criteria: Excellence, Impact, Quality and efficiency of the implementation.
The experts who have read the proposal congregate in Brussels (or Luxembourg) to come to a consensual view. The group prepares a Consensus Report CR.
Individual reading
Consensus Report - CR
As soon as one expert submits the IER, you will get an active task in SEP and the expert’s comments will become visible to you.
Still at home, you make a first draft of the Consensus report.
Remote phase: Draft CR, save it, don't submit
Start drafting the CR on basis of finished IER's
On-site phase: Consensus meeting, thereafter finalise the CR and submit
Consensus Report 1
If the IERs show a strong convergence of negative views, the Moderator may ask you to finalise and submit the CR without a formal meeting
Thereafter, the Moderator and the experts in the Consensus group will sign the form, if they agree. If any of them does not agree with your report, it will be sent it back to you with requests for changes
Consensus group
Working in the Consensus group
Consensus Report
In the draft CR that you may (not all topics request a draft CR) prepare before the presence meeting in Brussels or Luxembourg, make sure you set apart, by means of format (e.g. normal/italics, normal/bold, black/coloured), the experts' contentious and uncontentious points.
Don't "submit" this draft, just "save" it.
The Consensus group discusses the proposal until a common view is reached on comments and scores
The Commission Moderator chairs the group, ensures that all voices are heard and all issues discussed
You record the resulting comments and scores in the Consensus Report form in the SEP tool. Your work is projected onto a screen so that the entire group can see your progress.
When you are done and finished, please submit the Report in the tool.
Then, all experts and the moderator sign it. If any of them disagrees with the report, s/he may send it back to you with a request for redrafting.
Consensus group 1
The Commission Moderator actively seeks consensus among the experts. Please help by summarising, prompting, suggesting good wording, without, of course, expressing your own view
Please help the moderator bring out the differences so that they can be clarified and resolved
Indicate omitted bullet points (sub-criteria listed under each evaluation
criterion)
Help the Commission Moderator maintain equal judgment standards across all meetings
Consensus group 2
Comments must be factual, not unsupported opinions
Be aware of the danger of “dominating negatives” – do not forget to mention the proposal’s strong points, too!
Make sure your comments are really understood by everybody (especially those experts with poorer English)
Familiarise yourself with the scoring scale; make sure the scores match the set of comments!
Make sure each argument is put under the right criterion
Don’t appear to punish a proposal twice for the same "crime". A basic underlying fault in a proposal may impact more than one criterion, but in this case help make clear that the experts address different and distinct aspects under the different criteria
Consensus group 3
You are knowledgeable in information and communication technologies yourself….
… and you are aware of information which the experts do not seem to be taking into account Keep quiet, please! If you start contributing information you
may end up contributing an opinion! Discuss the matter afterwards with the Commission Moderator. If s/he agrees, the new piece of information will be brought up by the Commission chairperson at Panel stage
… and you see the experts making a factual error Speak up! The meeting can be briefly paused while the facts
are checked
Always finish by verifying that the experts agree with your CR and are okay with the outcome ! Specifically ask: "Are you in agreement with this CR?" "Is there anything else to add?"
If possible, you, the experts and the moderator sign the form in the SEP tool right at the end of the meeting
Consensus group 4
Consensus Report 2 The Consensus Report only gives the final agreed view on the
proposal, no intermediate steps
If scores 1 to 4 are given (there are shortcomings to a various extent), make sure that the shortcomings are named! But do not recommend proposal changes!
In the “Overall comments” box:
• You need not write anything
• For below-threshold proposals, you may write “This proposal scores below the evaluation threshold for criterion…” (BUT: No repetition of criterion comments! No additional discussions or remarks!)
• For high-scoring proposals – comment, for correction, on minor errors in the proposal (typos, wrong numbering …), or on any (rare) aspect not fitting under an evaluation criterion, but do not recommend changes
• For proposals with 3rd-country funding request, put down whether the 3rd-country applicant's contribution is essential for the work
Proposal Minute field 1
The Consensus report also has a "proposal minutes field" to be completed by you with the support of the Commission Moderator. Please put there a short record of the discussion's evolution towards the consensual view
Exceptionally, a meeting may end without consensus
among the experts. In this case, you put the majority view in the criteria boxes of the Consensus report, while the "minority-opinion" box of the consensus form details the minority view
Proposal Minute field 2
The Proposal Minute field is the Commission's 'memory' of the evaluation
In this field, please focus on the process, not the outcome
Record important discussion and crucial decision points
Especially, note down the reasons why an expert agreed to deviate starkly from his/her opinion in the IER
Quality check of the Consensus Report
After the Consensus Meeting, the Consensus Reports undergo a quality check performed either by Commission staff, or by a dedicated Recorder
Typos, duplicate words, layout problems etc. are cleaned out by this person
However, if there is a substantial problem with the text, e.g. a mismatch between comments and scores, the Report will be sent back to you with a request for correction. In this case, the Evaluators and the Moderator have to agree with and sign the Report again.
The Panel meeting
Working in the Panel meeting
The Panel meeting
In most cases, the Panel first approves the rejection of the below-threshold proposals
Then, each above-threshold proposal is briefly introduced by its Consensus-meeting Recorder
The Panel discusses and suggests any necessary modification to the proposal’s CR, to convert it to an Evaluation Summary Report (ESR). Modifications to the comments may be entered by you, or a dedicated Panel Recorder, changes to the scores by Commission staff
If there are any modifications to the Consensus comments or scores, the Proposal or Panel Recorder completes the "Proposal Minutes at Panel Stage" field, recording the motivation for the modification. If there are no changes, please minute “No changes”
The Panel report You may also be asked to assist in drafting the Panel report
• Stick to the standard template! Omit nothing, invent nothing, except for a column showing to which sub-topic each proposal belongs, if the WP foresees such sub-topics
• The report only describes the final result for each proposal. In particular, it never discusses any changes made in the Panel meeting
• Below-threshold proposals are rarely discussed, only listed in a table
• The panel report is signed on paper by the panel recorder plus at least the majority of experts present at the panel meeting
• If, at the end of the panel meeting, the panel report is not yet ready for signature, at least a majority of the experts sign the panel meeting minutes, which include the ranked list and the list of issues
You may additionally be asked to assist in preparing informal panel minutes
ICT evaluation
Writing a good
Consensus Report / ESR
Consensus report / ESR 1
• Comments are put under the pertinent criterion
• Comments are clear and substantial
• Comments are of adequate length
• Comments give facts, not opinions – “This proposal is.. .” not “We think that....”
• You identify the shortcomings, if any, but you do not make recommendations for changes, please
• There is no advice, please, in view of a proposal's re-submission!
• There is no identification of any evaluator
Consensus report / ESR 2
Poor comments don’t clarify the score – Good comments explain it!
• This proposal does not adequately advance the state of the art
• This proposal fails to advance the state of the art in X or Y, it does not take Z into account
• Good potential impact can be foreseen • The implementation plan addresses the two key
potential market segments A and B
• Planned resources are inadequate • Resources in the main research work packages 2 and
3 are underestimated by some 25%
Poor comments are shallow The objectives are not specified in terms of their clinical and S & T intent. No evident progress beyond state-of-the-art is given
Good comments are deep and cover all bullet points The proposal lacks a clear identification of the state-of-the-art, key technologies and design approaches. It does not sufficiently consider research in language learning and formal theories of knowledge. The S&T approach is unlikely to enable the project to achieve its objectives. The required foundational research is not well addressed. There are numerous statements which are not grounded on relevant research results. The proposal does not consider the use of standards nor does it contribute to their development.
Consensus report / ESR 3
Poor comments are ambiguous – Good comments are clear
• The resources for the project are unrealistic (Does this mean too high, or too low?)
• The overall resources are overestimated by 30%
• The composition of the consortium is not appropriate (Does this mean too many partners, or too few?)
• The Consortium does not show adequate expertise and resources in the area of X
Consensus report / ESR 4
Poor comments are vague, or suggest ignorance or uncertainty - Good comments are precise and final
• We think the consortium management plan is probably inadequate given the duration of the project and the number of partners
• The consortium management plan does not include an overall responsibility for demonstration activities; it omits a problem-solving mechanism in the event of disputes between partners
• The relevance of Workpackage 4 to the goal of the project is not clear to us
• Work package 4 is not relevant to the goals of the project because it fails to address issue A but instead dedicates the majority of its effort to B
Consensus report / ESR 5
Poor comments provide an opening for a complaint
• There is no discussion of dissemination activities • Dissemination activities are not adequately discussed
• There are too many Italian partners in the consortium • The project's effort and impact are over-concentrated in
one country or region
• There is only one end-user organisation in the consortium • The consortium lacks a sufficient participation of end users
• The coordinator is not adequately experienced • The coordinator does not demonstrate an adequate level of
experience of work in this field
Consensus report / ESR 6
Poor comments include words like: Perhaps
Think
Seems
Assume
Probably
Good comments include words like: Because
Percent
Specifically
For example
Consensus report / ESR 7
Start from the given vocabulary (“…poor, fair, good, very good, excellent…” ) and expand from there
Why say “Poor” when you can say:
Insufficient, minimal, fails to describe, unacceptable, inadequate, very generic, not evident, unfocused, very weak, bad, does not meet criteria, no information, inappropriate, limited, unclear, not sound enough, not specified, no significant impact, not been followed, unjustified, overestimated, does not fit profile…..
Why say “Excellent” when you can say:
Extremely relevant, credible, very clear, precisely specified, realistic, very innovative, extremely well suited, very good, timely, convincing, comprehensive, high quality, justified, very well identified, strong, highly effective, thoughtful, very promising, evidence, well-formulated, carefully-prepared, very professionally prepared, fully in line, looks great, very profound, sound, very convincingly integrated, clearly articulated, coherent, well balanced, very plausible, ambitious, clear advances, well above average……
Consensus report / ESR 8
Check your work
• Have you fully explained this proposal’s strengths and weaknesses against the evaluation criteria?
• Do scores match comments (high scores = more positive comments, low scores = more negative comments)?
• Have you checked any matters of fact, which you have quoted?
• Have you written comments at adequate numbers and length?
• Spelling (Spellchecker) and Grammar are correct?
Consensus report / ESR 9
If you were the proposer to receive this ESR, would you find it fair, accurate, clear and complete?
Last check
Where to find the recorded briefing and slide deck
• Slide deck and URL of the recorded briefing will be available in the SEP evaluation tool
• namely at the bottom of your DASHBOARD under Guidance/Supporting documents
Thank you for your attention! [email protected]
H2020-ICT-2019-2: URLs of the briefings
General expert briefing: https://webcast.ec.europa.eu/ict-2019-2-dt-2019-1-evaluatorswebcast
ICT-06-2019: https://webcast.ec.europa.eu/web-briefing-to-evaluators-and-rapporteurs-for-call-ict-06-2019-
unconventional-nanoelectronics
ICT-08-2019 https://ecwacs.webex.com/ecwacs/ldr.php?RCID=cf9ef9342d02ef7bab44f0e2cc4d301c
ICT-09-2019 & ICT-10-2019 https://webcast.ec.europa.eu/ict-2019-2-dt-2019-1-evaluatorswebcast (at 1 hour 30 min)
ICT-13-2019: https://ecwacs.webex.com/ecwacs/ldr.php?RCID=a8d718aa823131107595b6fedcc70faf
ICT-20-2019: https://ecwacs.webex.com/ecwacs/ldr.php?RCID=fb5b0f1dd9a5764c4a4fa1cf294a9013
ICT-24-2019 & ICT-31-2019 https://ecwacs.webex.com/ecwacs/ldr.php?RCID=7198483cce4ba15072126b28cf3f1af1
ICT-33-2019 https://ecwacs.webex.com/ecwacs/ldr.php?RCID=52ad2a026c1c82774fb240170aa0470b