PROPOSAL EVALUATION:

Guidance for evaluators

for Calls in FP7 (Framework Programme 7)

Sergey Mikhalovsky,

Nazarbayev University

My recent experience with research funding

Co-ordinator, Principal Investigator or Team leader:European Union: FP6-MATISS (2006-2010)FP7-MONACO-EXTRA (2008-2012) FP7-OncoNanoBBB (2012-2015) FP7-ABREM (2010-2014), FP7-FRESP (2009-2012)FP7-Greenland (2009-2012)Interreg IIIA Stent (2005-2008) Interreg IVA Flax (2009-2012)TEMPUS III and TEMPUS IV (2005-2008, 2009-2012)

UK: Technology Strategy Board - FullFlush (2010-2013), Department of Health (2008-2013), Medical Research Council (2012-2015)

My experience with proposal evaluation

FP5, FP6, and FP7 in nanotechnologies, environmental sciences and Marie Curie Programme

Erasmus Mundus I and II, Lead Expert in Life Sciences

TEMPUS II and III, INTAS, EPSRC (UK), BBSRC (UK), national programmes for Russia, Austria, Cyprus, France, USA and Kazakhstan

FP7 Specific Programmes

Co-operation – Collaborative Research

European Research Council

Marie Curie Actions

Capacities – Research Capacity

JRC EURATOM

Co-operation

Ideas

People

Capacities

JRC

Budget Split 2007-2013

€4 700

€7 460

€4 097

€1 751

€32 413

Values in € Millions

What about Kazakhstan?

Kazakhstan is on the list of International Cooperation Partner Countries (ICPC)

And eligible for most FP7 projects!

7

Rules on submission & evaluation

Basic principles Excellence. Projects selected for funding must

demonstrate a high quality in the context of the topics and criteria set out in the calls.

Transparency. Funding decisions must be based on clearly described rules and procedures, and applicants should receive adequate feedback on the outcome of the evaluation of their proposals.

Fairness and impartiality. All proposals submitted to a call are treated equally. They are evaluated impartially on their merits, irrespective of their origin or the identity of the applicants.

8

Rules on submission & evaluation

Basic principles Confidentiality. All proposals and related data,

knowledge and documents communicated to the Commission are treated in confidence.

Efficiency and speed. Evaluation, award and grant preparation should be as rapid as possible, commensurate with maintaining the quality of the evaluation, and respecting the legal framework.

Ethical and security considerations: Any proposal which contravenes fundamental ethical principles, or which fails to comply with the relevant security procedures may be excluded at any time from the process of evaluation, selection and award

9

The Three MainReference Documents

1 - Rules on submission and evaluation This is the common and official reference for FP7 rules for

submission of proposals, and the related evaluation, selection and award procedures.

2 - Guide for applicants The Guide for Applicants contains the essential information to guide

proposers through the mechanics of preparing and submitting a proposal.

All proposals shall contain a Part A (administrative forms) and a Part B (proposal description). Indications about the content and issues to be addressed are described in the Guides for Applicants

Please make sure that you read the “Guide for applicants” that corresponds to the funding scheme for the topic.

10

The Three PrincipalReference Documents

3 - The work programme: three complementary documents:

Work Programme - General Introduction

2012 Work Programme: includes the topic’s description and criteria against which the proposals will be assessed.

Cooperation" Work Programme - General annexes Annex 1: List of International Co-operation Partner Countries (ICPC) Annex 2: Eligibility and Evaluation Criteria for Proposals Annex 3: Forms of Grant and Maximum Reimbursement Rates for Projects

Funded Annex 4: General activities

11

Example: Joint call ENV-NMP

Work Programme – Topic descriptionNanotechnologies for water treatmentNanotechnology presents many benefits for environmental technology applications, such as remediation, treatment or sensor development and monitoring purposes. In the field of water, nanotechnology has the potential to contribute to long-term water quality, availability, and viability of water resources such as through advance filtration that enables sustainable water reuse, recycling or desalination. The aim of this action is to support research and technological development in the field of water treatment by applying developed or adapted nano-engineered materials to promising separation, purification and/ or detoxification technologies.

Proposals should focus on process intensification aiming at improving selectivity, robustness, stability and performance while reducing energy requirements and by-product generation. Specific monitoring issues, as well as safety, environmental and health aspects, should be included if directly associated with the new technological solution proposed. Priority will be given to novel ideas and emerging technologies promising major advances and a large potential impact in the long-term, including cost-effectiveness

12

Joint call ENV-NMP

Work Programme – Expected Impacts

Development and uptake of innovative and cost-efficient water treatment technologies benefiting from progress and advances made in nanosciences, materials and technologies.

This initiative should deliver step-change advances in water treatment technologies, including validation and verification of arising prospects in terms of improving treatment performance and reducing energy requirements.

By fostering the knowledge base in this area, the projects addressing this topic will contribute to strengthening European competitiveness in the water sector and the implementation of the Environmental Technologies Action Plan and the Nanotechnology Action Plan.

13

IndependentIndependent expertsexperts

Expert evaluators are at the heart of the FP7 system Provide independent, impartial and objective advice

to the Commission they represent neither their employer, nor their country!

Significant funding decisions will be made on the basis of their advice They can also add value to projects through your comments

and suggestions

The integrity of the process is crucial They should follow the Code of Conduct annexed to the

appointment letter

Who evaluates? Selected from a wide pool from a database, on the

basis of keywords Minimum 3 evaluators per project Selected per call Replace about ¼ in any given area annually Sign confidentiality and conflict of interest

declarations Names published after evaluation (though not at call or

proposal level) Target at least 40% female Mix of geographical location and background To register:   https://cordis.europa.eu/emmfp7

FP

7 –

How

to

appl

y

15

Overview of the Evaluation ProcessOverview of the Evaluation Process

Full Proposal

Proposalforms

Evaluators

Eligibility

Evaluators Evaluators Final rankinglist

PanelSubmission ConsensusIndividual

reading

Proposals insuggestedpriority order

Rejection list

Finalisation

CriteriaCriteria Criteria

COMMISSION COMMISSION

“rem

ote”

may

be

“rem

ote”

Role of experts

16

ProcessFor each proposal:

Proposal Xcopy 1

Proposal Xcopy 2

Proposal Xcopy 3

IERexpert 1

IER expert 2

IER expert 3

Consensus meeting

CR 3 experts

Note: There may be more than 3 evaluatorsIER=Individual evaluation reportCR=Consensus Report

“remote” May be “remote” and / or central

17

ProcessEthical Issues

The Consortium is asked to submit drafts of Information Sheet and Consent Form but does not need to submit copies of legislation

Proposals should comply with fundamental ethical principles relevant security procedures

… or be excluded from the process Check if the proposal has in fact ethical issues

If yes: tick ethical issues box in the CR Prepare Ethical Issues Report

18

ProcessEvaluating a proposal

Three guiding principles: Objectivity

each proposal is evaluated as it is written Accuracy

The judgment is made against the official evaluation criteria, and nothing else

Consistency The same standard of judgment applies to

each proposal

19

ProcessThe evaluation criteria

Criteria adapted to each funding scheme and each thematic area specified in the work programme

Three main criteria: S&T Quality (relevant to the topic of the call)

Concept, objective, work-plan Implementation

Individual participants and consortium as a whole Allocation of resources

Impact Contribution to expected impacts listed in work

programme Plans for dissemination/exploitation

20

ProcessFP7 Evaluation Criteria

Applicable to ALL funding schemes1. S/T quality

(in relation to the topics addressed by

the call)

2. Implementation 3. Impact

Sound concept, and quality of objectives

Appropriateness of the management structure and procedures

Quality and relevant experience of the individual participants

Contribution, at the European and / or international level, to the expected impacts listed in the work programme under the relevant topic / activity

21

ProcessFP7 Evaluation Criteria

Collaborative projects1. S/T quality

(relevant to the topics addressed by the call)

2. Implementation 3. Impact

Progress beyond the state-of-the-art

Quality and effectiveness of the S/T methodology and associated work plan

Quality of the consortium as a whole (incl. complementarity, balance)

Appropriate allocation and justification of the resources to be committed (budget, staff, equipment)

Appropriateness of measures for the dissemination and/or exploitation of projects results, and management of intellectual property.

22

Collaborative projects

Support to research projects carried out by consortia with participants from different countries, aiming at developing new knowledge, new technology, products, demonstration activities or common resources for research.

The size, scope and internal organisation of projects can vary from field to field and from topic to topic.Projects can range from small or medium-scale focused research actions to large-scale integrating projects for achieving a defined objectiveProjects may also be targeted to special groups such as SMEs, Specific International Co-operation Actions, etc.

Funding schemesFunding schemes

Process

23

International cooperation

Some topics in work programme seek participation from third countries International Cooperation Partner Countries (ICPC) industrialised countries

research partners from ICPC may be financed

24

ProcessProposals that are only

partly in scope

Note: The “S/T quality” of a proposal (first criterion) is evaluated to the extent that the content is relevant to the topic(s) addressed by the call E.g. If a proposal is only marginally relevant, or if

only one work package is relevant, the evaluator must downgrade the score – no matter how excellent is the science!

Relevance to the objectives of the call is also considered under “Impact” (third criterion)  In relation to the sub-criterion “contribution to

expected impacts listed in the work programme”

25

ProcessProposal scoring:

Interpretation of the scores 0: The proposal fails to address the criterion under

examination or cannot be judged due to missing or incomplete information

1: Very poor. The criterion is addressed in a cursory and unsatisfactory manner.

2: Poor. There are serious inherent weaknesses in relation to the criterion in question.

3: Fair. While the proposal broadly addresses the criterion, there are significant weaknesses that would need correcting.

4: Good. The proposal addresses the criterion well, although certain improvements are possible.

5: Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor.

26

ProcessProposal scoring

Each criterion is scored 0-5 Marks can go from 0 – 5 in steps of 0.5, i.e half-marks are allowed Experts are encouraged to use the whole range Scores must pass thresholds if a proposal is to be considered for funding

Thresholds apply to individual criteria… Threshold is 3

…and to the total score higher than the sum of the individual thresholds Threshold is 10

Note that to receive a mark of 5, a proposal does not have to be perfect. An excellent proposal can have minor shortcomings.

When writing comments in the IERs and Consensus Report, the severity of any weakness should be clearly stated, i.e. are they minor, moderate or significant

27

ProcessProposal scoring

Evaluate the proposal and conclude whether the proposal is Excellent Good Fair Poor Very poor Failing to address the criterion

Score the proposal accordingly

28

ProcessCommission Follow-up

Evaluation summary reports sent to applicants “initial information letter” Redress procedure

Draw up final ranking lists Information to the Programme Committee Contract negotiation Formal consultation of Programme Committee (when

required) Commission decisions Survey of evaluators Independent Observers’ reports

New for FP7

29

ProcessRedress

Proposers can complain if they believe there have been shortcomings in the handling of their proposal, and that these shortcomings have jeopardised the outcome of the evaluation process.

The quality and consistency of the evaluation reports (ESRs), derived directly from the CRs, is paramount to minimise the redress procedures

An interesting and innovative idea Strong consortium Experienced co-ordinator Complementarity of skills and expertise Perfect matching between the proposal and the

call At least 6 months to prepare a proposal European added value Sustainability after the end of the project Dissemination strategy

FP

7 –

Tip

s

Tips on writing a successful proposal

The EC Funding Process –A Rough Guide to :

Why, What and How

Sergey Mikhalovsky

Why?

Good Money – typically 200,000-300,000 € per participating organisation for 2-3-4 years

More if you are the Lead Partner – all the money is distributed through you, which counts for your organisation – good for you and for your organisation – typically 1-2-3 million € for 2-3-4 years

Good Value for Money – typically 60% overheads

Why? (continued)

Fun – experience of different countries, cuisines and cultures, meeting new people

Knowledge transfer – opportunity to work in different labs, access to unique equipment and instruments, methods and techniques

Human mobility – you visit others and others visit you to exchange knowledge and experience

Fair competition and useful feedback

Why? (cont.)

Networking – you build up your own network There may be unique expertise somewhere in

Europe unavailable in this country Career enhancement Publicity – EC loves it and gives you lots of

opportunities to do that (even if you don’t want it...)

Recruitment of high quality researchers And you may even get a result...

What?

Incredibly diverse formats and degrees of participation:

(i) individual fellowships – No age limit, very well paid!

(ii) participation as a team – you can be paid or you can employ someone

(iii) co-ordinator – the same as (i) and (ii) plus management costs plus permanent headache.

What? (cont.)

Funding is available for salaries, overheads, travel, consumables, management, equipment and organisation of networking events.

How?

37

Regularly monitor the funding opportunitiesMost obvious source of funding for regular monitoring are: (i) FP7 - become a regular visitor on this site: http://ec.europa.eu/research/participants/ portal/page/home and become familiar with the structure of this programme. It is the main but not the only source of R&D funding in Europe

How? (cont.)

(i) Be flexible with your idea of participation – sometimes there may be a Call which matches exactly what you want to do but that is unlikely.

(ii) Identify potential areas of interest by monitoring previous Calls and - important! – analysing success rate in previous Calls. If the success rate is less than 10% - don’t go for it, unless you are a winner in a National Lottery or somebody else is prepared to do the main job of writing for you.

How? (cont.)

Try to get information about forthcoming Calls – it is possible to do so by attending consultations in Brussels or elsewhere, contacting your NCP (National Contact Point), getting information from an insider, etc.

The earlier you know the contents of the future Call, the better your chances for success. When the Call has been announced, it is too late! You will have 2-3 months to submit a proposal from the date when the Call opens.

How ? (cont.)

There are also bottom-up Calls – these are my favourite – you can write about anything you like and the date of the new Call announcement is usually known well ahead, so you have plenty of time to think and prepare for the Call.

How ? (cont.)

There are also bottom-up Calls – these are my favourite – you can write about anything you like and the date of the new Call announcement is usually known well ahead, so you have plenty of time to think and prepare for the Call.

Erasmus Mundus and TEMPUS programmes offer in essence bottom-up Calls in Research&Education.

What to start with?

Formulate the idea, which fits a particular Call (probably several ideas to have a choice)

Identify potential partners – do a lot of networking, and do it all the time regardless of Calls.

Learn the Brussels speak – it takes time, but once you understand it – your chances for success are much higher.

What to do next?

You have to understand how to write a proposal

Become an expert evaluator for FP7 https://cordis.europa.eu/emmfp7/ ?fuseaction=wel.welcome

It will help you to understand the evaluation process and ultimately improve your proposal writing skills

What to do next?

Write a proposal – preferably not by yourself if it is your first one.

Write a proposal yourself if you have done this before

Seek advice and assistance of people with experience: colleagues who have got such grants, Research Office

Interact with other partners Notify your line managers and finances about

your intention to submit a proposal in advance

S&T Quality of the proposal Concept and objectives. What are the main

ideas that led you to propose this work?

Provide sufficient elements on the planned S&T methodology and describe in detail the S&T objectives. Show how they relate to the topics addressed by the call, which you should explicitly identify. The objectives should be those achievable within the project, not through subsequent development. They should be stated in a measurable and verifiable form, including through the milestones and deliverables.

45

S&T Quality (cont.) Progress beyond the state-of-the-art. Describe the state-of-the-art in the area

concerned, and the advance that the proposed project would bring about. If applicable, refer to the results of any patent search you might have carried out.

S/T methodology and associated work plan. A detailed work plan should be presented, which should follow the logical phases of the implementation of the project.

46

S&T Quality (cont.) Describe the overall strategy of the work plan. Show the timing of the different WPs and their components Deliverables and milestones Describe any significant risks, and associated contingency

plans Milestones are control points where decisions are needed

with regard to the next stage of the project. For example, a milestone may occur when a major result has been achieved. Another example would be a point when the consortium must decide which of several technologies to adopt for further development. Intangible – something which cannot be touched.

Deliverables: report, prototype, material, etc. Tangible - something which can be touched.

47

Implementation

Management structure and procedures

Describe the organisational structure and decision-making mechanisms of the project.

Show how they are matched to the complexity and scale of the project.

Individual participants.

provide a brief description of the legal entity, the main tasks they have been attributed, and the previous experience relevant to those tasks. Provide also a short profile of the staff members who will be undertaking the work.

Implementation Consortium as a whole.

Describe how the participants collectively constitute a consortium capable of achieving the project objectives, and how they are suited and are committed to the tasks assigned to them. Show the complementarity between participants. Explain how the composition of the consortium is well-balanced in relation to the objectives of the project.

Resources to be committed.

Show how the resources will be integrated in a coherent way, and show how the overall financial plan for the project is adequate. Infrastructure.

49

ImpactWhat does your project bring to:Individuals involved in the project;Teams involved in the project;Consortium as a whole;Region, country as a whole; international dimensionNew IP – profitSocio-economic benefits, environmental significance, healthDevelopment, dissemination, use of results

50

Impact (cont.)

Dissemination and/or exploitation of project results, and management of IP.

Describe the measures you propose for the dissemination and/or exploitation of project results, and how these will increase the impact of the project. You should take into account a variety of communication means and target groups as appropriate (e.g. policy-makers, interest groups, media and the public at large). With regard to the innovation dimension, where appropriate, describe the potential areas and markets of application of the project results and the potential advantages of the resulting technologies/ solutions compared to those that are available today.

51

Impact (cont).

With regard to the innovation dimension, where appropriate, describe the measures you propose to increase the likelihood of market uptake of project results, such as: verification,

testing, and prototyping; supporting the development of technical standards; identifying and collaborating with potential users; identifying potential partners and sources of finance for commercialisation.

Describe also your plans for the management of knowledge (intellectual property) acquired in the course of the project.

Networking

additional skills (i.e. project management, drafting of project proposals, language courses, ethics, IPR, CV writing, job search, interview skills, etc).

consider wider disciplines and all potential research environments (academic/industrial).

“hands on” experience.

→ future employment prospects

Training

Dissemination

attendance at international conferences and workshops

organisation of final conference with wider participation or satellite workshop at major conference

dissemination of research undertaken send documentation of success stories to Commission for further dissemination

Ethics

Describe any ethics issues that may arise in the project. In particular, you should explain the benefit and burden of the experiments and the effects it may have on the research subjects.

All funded research must comply with the relevant national and international ethics related rules and professional codes of conduct.

Where necessary, the beneficiary shall provide a written confirmation that (a) favourable opinion(s) of the relevant ethics committee(s) has (have) been received and, if applicable, the regulatory approval(s) of the competent national or local authority(ies), before beginning any research requiring such opinions or approvals.

56

What (not) to expect?

Expect:

Your first attempt will be a failure

Your second attempt may or may not be successful

One day you will succeed!

Do not expect:

Somebody else will do it for you!

Case Study Individual Incoming Fellowship (from 3rd country to UK)

submitted in 2008 (unsuccessful) and 2009 (successful) Criterion 1: Scientific Quality

Issues to be addressed when assigning an overall mark for this criterion: Scientific/technological quality, including any inter-disciplinary and multidisciplinary aspects of the proposal;

Research methodology; Originality and innovative nature of the project, and relationship to the 'state of the art' of research in the field; Timeliness and relevance of the project; Host scientific expertise in the field; Quality of the group/researchers in charge

2008: Solid phase hydrosilylation conditions are not defined.

It is not clearly defined how the synthesized products would be characterized. The novel materials that are claimed to be the outcome of the proposed project are too vaguely defined.

2009: no weaknesses!58

Case Study (cont.)

Criterion 2: Transfer of knowledge

Issues to be addressed when assigning an overall mark for this criterion:

Potential of transferring knowledge to European host and/or bring knowledge to Europe; Clarity and quality of the transfer of knowledge objectives.

2008: The knowledge that the applicant will bring to the host institute has not been sufficiently defined.

2009: no weaknesses!

59

Case Study (cont.)

Criterion 3: Quality of the researcher Issues to be addressed when assigning an overall mark for

this criterion: Research experience; Research results including patents, publications, teaching etc.; Independent thinking and leadership qualities, and capacity to transfer knowledge; Match between the fellow's profile and project

2008: The fellow has a relatively modest publication activity. No publications in high impact journals have been presented. It is therefore difficult to judge the quality of the applicant's research from the information provided. It is not clear from the project description what characterization techniques the fellow used in her previous investigations.

2009: no weaknesses!60

Case Study (cont.)

Criterion 4: Implementation Issues to be addressed when assigning an overall mark for

this criterion: Quality of infrastructure / facilities and inter-national collaborations of host; Practical arrangements for the implementation and management of the scientific project; Feasibility and credibility of the project, including work plan; Practical and administrative arrangements, and support for the hosting of the fellow.

2008: Training on new research equipment has not been sufficiently discussed.

2009: no weaknesses!

61

Case Study (cont.) Criterion 5: Impact Issues to be addressed when assigning an overall mark for this

criterion: Potential for creating long term collaborations and mutually beneficial co-operation between Europe and the third country; Contribution to European excellence and European Competitiveness; Contribution to the socio-economic development of the Developing Countries or emerging and transition economies by transfer of knowledge and human capacity building (where relevant); Benefit of the mobility to the European research area

2008: In the case of negative results the contribution to European excellence and European competitiveness will be low. There is no risk analysis.

2009: no weaknesses!

62

Case Study (cont.)

Overall comments: 2008: The overall impact of this project, if

successful in all aspects, would be high, with pronounced contribution to the European excellence and European Competitiveness in an important scientific field. However, the associated risks have not been properly treated in the proposal.

2009: no comments!

63

Case Study (cont.)

Score 2008 2009S&T quality (3) 4 4.8

Transfer of knowledge (0) 3.8 4.8

Quality of the researcher (4) 3.5 4.9

Implementation (0) 4.6 4.9

Impact (0) 4.2 4.6

Total: 78.6 96.0

(in brackets – threshold mark) (failed) (awarded)

64

Some observations from 2011 evaluation

The aim of this project is a development of fundamental scientific principles of the obtaining of nanocomposite materials based on metal nanoparticles stabilized with polymeric matrices, a study of their structure, mechanisms of formation and electrocatalytic activity in the processes of hydrogenation of organic compounds, as well as nanocomposites from liquid crystal compounds and carbon nanotubes and a study of their structural features and physicochemical properties.

Your comments, please!

65

Analysis of the previous slide

Too verbous; difficult to measure, it is a mixture of aim and objectives, unclear what is this for?

Actually, I gave it 7 out of 9 in S&T criterion!

66

Expected results

Not quantifiable parameters; ‘Designed magnetic nanotechnology will

enable a new way to solve many scientific-technical and medico-biological problems’.

67

Expected resultsБудет разработана технология нанесения отдельных пленок Ag, Cu, также пленок содержащих оба этих металла, что расширит диапазон практического применения бактерицидных изделий. По этим технологиям будут выданы исходные данныe по проектам получения бактерицидных изделия бытового и медицинского назначения, а также документации по эксплутационным характе-ристикам этих изделий. Практическая значимость заключается также в том, что в предлагаемых технологиях используется отно-сительно дешевая стандартная аппаратура. Это будет способство-вать ее применению на предприятиях малого и среднего бизнеса.

Научная значимость результатов исследования заключается в том, что для получения пленок используется низкотемпературная реакция восстановления в газовой фазе. Причем восстанавливае-мые соединения могут находиться как в растворе, так и в твердой фазе. Благодаря этому можно добиться образования пленки и на внешней поверхности изделия и на внутренних поверхностях отдельных пор. Это позволяет добиться хорошей адгезии получаемых пленок. Кроме того это создает предпосылки для применения этого процесса в других технологиях.

68

Expected results - comments

The expected results are described vaguely; not clear what products will be developed and why the products developed will be better than those existing on the market. There are no target properties given for the products.

Not a single quantitative parameter is mentioned!

69

Research facilities Проект будет выполнен силами сотрудников с использованием собственного оборудования. Нормативно-методическая обеспе-ченность будет осуществляться отделом метрологии и стандар-тизации. Патентно-лицензионная сторона проекта будут обеспе-чиваться патентным отделом, который функционирует в составе научно-исследовательского отдела, а также образовательно-ин-формационным отделом, подключенного к Интернету. Приборы, используемые в выполнении эксперимента, прошли метрологичеc-кую проверку, аттестацию и аккредитацию. Исследования физико-химических свойств продуктов будут проводиться в аккредитованном испытательном центре XXX. Имеются 3 специа-лизированные лаборатории площадью более 200 м2 с подведен-ными коммуникациями. Лаборатории оснащены основным и вспо-могательным оборудованием для проведения исследований. Опытно-промышленная база для выпуска опытной партии про-дукта может быть создана в проблемной лаборатории, материаль-но-техническая оснащенность и производственные площади которой позволяют разместить все технологические линии.70

Research facilities - comments

The description of the available equipment is inadequate; some new equipment is planned for acquisition.

there is no description of the required manpower and skills.

71

References1. Обзор рынка редкоземельных элементов в СНГ. - М.: Инфомайн. 2008. - 101 с.

2. Михайличенко А.И., Михлин Е.Б., Патрикеев Ю.Б. Редкоземельные металлы. М.: Металлургия. 1987. 232 с.

3. Серебренников В.В. Химия редкоземельных элементов. - Томск.: Томский государственный университет. 1959. Т. 1. 521 с; 1961. Т.2. 802 с.

4. Химия редких элементов. Соединения редкоземель-ных элементов. Под ред. И.В. Тананаева. – М.: Наука, 1983. – 392 с.

5. Бюллетень иностранной коммерческой информации. 2007. №120

6. Информационные сообщения Гиредмет по материалам Industrial Minerals. 2002. April. С. 52-61.

72

References - Comments

Very basic, mostly in Russian and most out-of-date. Only one reference to an original publication in a peer-reviewed journal. The more recent references are to commercial reports rather than scientific or patent literature.

73

S&T common errors Lack of details in describing methodology; Generic words instead of concrete

examples;

74

There is always somewhere funding waiting for you and your idea(s).

You can do it

If not you – who else?

Questions?

GOOD LUCK!

Conclusions