Evaluation in R&D sphere in Ukraine: Real practice and

problems of transition to new standards

Igor YegorovCentre for S&T Potential and Science History Studies, National Academy of

Sciences of Ukraine and the Institute of Statistics, State Service of Statistics of

Ukrainee-mail: igor_yegorov1@ukr.net

Paper presented at the International workshop

Science evaluation as a prerequisite for promoting excellence in research

6 November 2012

Chisinau, Moldova

Key objects of evaluation in R&D and innovation sphere in Ukraine

a) Individual researchers

b) Research organizations (institutes, design bureaus and so on)

c) Research and innovation projects

d) R&D programs and R&D sections of development programs

Main sources of data

Expert opinion (individual experts or expert panels) State statistics (State Service of Statistics of Ukraine) Department of Intellectual Property Institute of S&T and Economic Information Internal statistics of different ministries and agencies (for

instance on functioning of technoparks) Information from special surveys and databases Foreign sources (internationally recognized publications,

awards and so on)

Problems with utilization of official statistics

Rigidity in approaches and difficulties in changes of indicators and questionnaires

Deviation from international standards (examples: system of scientific degrees, calculation of number of publications and so on)

The willingness to serve ‘political’ interests (example: number of research projects and other indicators, which are working ‘in favour’ of old system)

Key regulatory acts

Law on Scientific and S&T Activities (1992) Law on Innovation Activities (2002) Law on S&T Expertise (2006) Law on Higher Education (1993) Law on State S&T Programs (2004) Total number of laws and other regulatory acts on R&D

and innovation exceeds 80. Almost all of them contain clauses on evaluation at different levels.

Important features of the Ukrainian research system

High level of centralization in Ukrainian R&D system: administrative power of rectors and directors of the research institutes is ‘excessive’ (including full control over all financial transactions)

System of the evaluation of research results with clear domination of internal publications, not international publications as indicators of success

The procedures of evaluation and selection of R&D projects are often not transparent and fair for potential participants.

Recent changes in R&D and Innovation statistics

Introduction of the CIS-type survey in Ukraine (2009-2010, data for 2008-2009). New survey is expected in 2013

Switching (partial) to international classification of sectors - since 2009

Implementation of new NACE-type classification (new version of KVED) – since 2012

Implementation of indicators of international publications in some universities and research institutes

Publications of Ukrainian authors in scientific journals in 1996-2005, tous. (Thomson

Reuters)

3700

3800

3900

4000

4100

4200

4300

4400

4500

4600

4700

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

Number of scientific publications according to the State Statistical Service of Ukraine,

1991-2011, thous.

Evaluation of individual researchers

Method: Panels of experts from the same institute (as a rule: at the level of department and at the level of the institute) discuss the work of researcher and vote openly for the decision. This decision has to be confirmed by the scientific council of the institute.

Periodicity: Every five years Indicators are used indirectly (number of publications,

participation in conferences and seminars, awards, preparation of the thesis (if any), participation in a work of different scientific councils and commissions and so on)

Evaluation of research organizations

A) Panel of external experts from the same research area prepares a report on activities (every four-six years). Indicators include number of publications, number of patents and so on.

B) Evaluation with implementation of the special metrics (2012)

Evaluation of potential (within the projects or programs) varies according to the goals of evaluation (academic programs, STCU and so on)

Evaluation with implementation of the special metrics (2012)

‘Objective’ indicators (60% of final assessment) Expert evaluation (40% of final assessment) ‘Objective’ indicators:

a) personnel (5 indicators)

b) financial situation and research equipment (8 indicators)

c) research in priority areas (3 indicators)

d) publication activities (2 indicators)

e) innovation activities (4 indicators)

f) international activities (3 indicators)

g)’visibility’ at the international level (2 indicators)

Example1: ’Visibility’ at the international level

Number of references on the research organization in the Google searching system

Number of publications in domestic and international scholar journals with high impact-factor per one employee of the research organization

Example 2: Publication activities

Number of publications in domestic scholar journals per employee of research organization

Number of publications in foreign scholar journals from international databases of scientific publications per employee of research organization (data bases include information from Scopus and some other databases on medical publications)

Expert evaluation of research organization

Level of research activities Relevance to the national priorities Potential for further development Indicators, which reflect specific features of research

area

Note: all indicators have their own weights, depending on the type of organization

Evaluation of research projects Ex- ante evaluation Evaluation of the process (if the project has long duration

or it is divided into several stages) Ex-post evaluation Different types of indicators are used, depending on the

type of the project Special forms are used by different agencies and

ministries, not less than two experts take part in the evaluation procedure

Key idea is to reach the goal of the project (must include, if possible, description of quantitative indicators, which have to be achieved) within certain time and financial constraints

Example of indicators for selection of the project for the Ministry of Economic

Development and Trade (October 2012) Proposed price (60% of the final assessment) Other indicators (40%):

a) Number of Doctors and Candidates of Sciences in the research team (30 points, if more than 7: 3D+4C)

b)Number of articles in scholar journals (30 points, if more than 10) on the topic of the project

c) Monographs on the subject (15points per monograph)

d) Participation in similar projects with the Ministry during the last 3 years (10 points per project)

Evaluation of the programs In accordance with to the normative acts and existing

laws at the state and ministry level Problems:

1. Lack of quantitative criteria and utilization of ‘artificially constructed criteria’

2. Non-transparent expert procedures (as a rule, no foreigners take part in the evaluation, excluding Key Laboratory initiative and some joint programs with foreign partners)

3. Lack of money for financing of R&D programs, which is used as an argument for justification of the level of results achieved within the program

Conclusion Ukraine has not yet created relevant institutions and

proper instruments necessary for effective transformation of R&D sector, including effective system of research assessment. Internal practice of evaluation differs substantially from the international one.

Unfortunately, every year the possibilities for the implementation of effective transformation policy are shrinking, as the number of researchers is declining and the research centres are losing their ability to conduct research.

Conclusion With time running out, remaining research institutions will

face more problems with the creative capabilities needed to compete internationally

Reforms of evaluation of R&D sector have to be an integral part of general transformation of R&D system

EU could contribute to the reforms by stimulating introduction of new standards of R&D project assessment and implementation

Thank you for your attention !