evaluation and benchmarking of rtd organisations - experience from regpot steering platform on...
TRANSCRIPT
Evaluation and Benchmarkingof RTD Organisations -
Experience from REGPOT
Steering Platform on Researchfor Western Balkan Countries
Zagreb, 29 October 2009
Evaluation Pyramide
many many many many many many many many manymany many many project evaluations
some RTD programme evaluations
just a fewevaluations of
RTD institution
almost no systemevaluations
The different levelsrequire different evaluationskills and methods
Peer Reviews
Bibliometrics, network
analysis etc.
Peer Reviews, SWOT, CBA
multi-level approaches
,etc.
Case Study
FP7-REGPOT-2008-2
SWOT-CHEMISTRY-FOOD (Evaluation of the research capacity and development of a strategy for further growth in chemistry in general and in food science in particular)
University Ss. Cyril and Methodius, SkopjeInstitute of Chemistry (around 50 personnel)
Target Group(s)
1. The management of the research organisation (institute or faculty or university)
2. (The employees of the institute - indirectly affected)
3. The policy-delivery system
RTD managers and policy makers are in the same boat:
Managers – to improve the competitiveness of the institute
Policy makers – to improve the quality of the entire innovation system
Organisations are important elements of the innovation system
The generic model of national innovation systems as presented in OECD (2003)
Policy Objectives
1. To map competitive excellence, resources and competencies and obtain transparent information about the state-of-art of the RTD organisations
2. To improve the performance of RTD organisations and, thus, to improve regional, national or local research and/or innovation systems
3. To have a evidence-based ground for channeling of funds and allocation of resources and factors
4. (to increase the application of a modern management tool in order to foster reflection and understanding over assumptions and beliefs)
A lot could be done, but it needs the power of a mammoth!
Evaluation Functions
1. Learning (information and steering function)
2. Accountability (legitimization and control function)
Target groups may have contradictory interest or not.
„Why evaluation?“
Case Study – Law on Scientific Centre of Excellence (1)
Scientific Centre of Excellence (Art. 23, 47, 48)
Article 23 (DEFINITION)A Scientific Center of Excellence is a scientific institution or a group of researchers who by merit of their originality, significance and relevance of the achieved results based on their scientific research activity belong to the institutions or groups of researchers with highest quality in the world in their area of scientific research.The Scientific Center of Excellence should have accomplished: a strong connection with learning, research and innovations, that is close relationship between research and industry; a focused research program which involves interdisciplinary topics, applicative and fundamental endeavors; to be able to provide a dynamic, flexible and attractive work environment which will result in high accomplishments, at an individual or team level; procedures to educate new generations of scientific and technological talents; as well as the capacity to accomplish dynamic partnership with the organs of the central and the units of local government.
Case Study – Law on Scientific Centre of Excellence (2)
Evalution of the Scientific Centre of Excellence (Art. 23, 47, 48)
- Subordinated to continual monitoring and external evaluation (efficiency, sustainability, efficacy and relevance of activities )
- Evaluation results will be published transparently. - Obligation to perform self evaluation of the quality of their
scientific research activity based on their own system for quality control management in accordance with their statute and the international standards for evaluation.
- The external evaluation is performed exclusively based on international standards by the Agency for Evaluation in accordance with the Law for Higher Education.
Evaluation of the Institute of Chemistry
Mode: voluntarily commissioned independent external evaluation carried out by two independent evaluators proposed by EC (Prof. Stane Pejovnik, Rector of the University of Ljubljana and Klaus Schuch, Director of ZSI)
Context: supported by an EC funded REGPOT project integrated in an overall evaluation of the food sector
Budget: € 54.000,00 out of which € 27.000,00 for external evaluators
Duration: € 1 year
Outputs: evaluation methodology (including questionnaires), evaluation report, action plan
Methodology
Borsi, B., Dévai, K., Papanek, G. and Rush, H. (2004): The RECORD Manual: Benchmarking Innovative Research Organisations in European Accession Countries. ISBN 92-894-7916-7European Commission, RTD info EUR 21238
3 Foci of a modern RTD organisation:- Knowledge generation- Knowledge diffusion- Knowledge utilization
SWOT-Factors:Different factors (categories) for the different foci- Internal factors (strengths and weaknesses)- External factors (opportunities and threats)- Negotiated factors
Data needs
input data (human resources and material and non-material investments)
output data (publications and citations are not enough!) information about framework conditions (organisational
framework conditions and systemic RTDI conditions) – they have a high influence (corridors of action and path dependencies; social and „cultural“ issues)
Internal Factors (1)
(1) Critical mass (size)
(2) Progressive management
(3) Good Human Resource management
(4) Output of a creative and innovative team
ad 1) Researchers (skill differentiation), support staff Infrastructure (scientific, generic technical, library facilities) R&D investments and access to external RTD infrastructures
ad 2) Availability of a strategy (user orientation, problem solving, how to
producte value added, resource development orientation etc.) Leadership, quality management, knowledge management, sales
management, IPR management, PR …
Internal Factors (2)
(3) Good Human Resource management
(4) Output of a creative and innovative team
ad 3) Training and staff development (vocational education, skills forecast,
promotion plans, employee performance, teambuilding measures etc.)
Diversity structure of staff (mentoring systems, gender and age structure, role models, foreign researchers, in- and outflow, trans- and interdisciplinarity of staff etc.)
Flexibility (communication culture, information infrastructure, team compositions etc. - an organisation is flexible if it has the capacity to adapt to new internal or external environments and to change in accordance with emerging organisational needs)
Internal Factors (3)
(4) Output of a creative and innovative team
Ad 4) contribution to product innovations (or product changes)? contribution to process innovations? contribution to service innovations? Patenting and licensing Bibliometrics (publication and citation counts) Research projects (collaborative projects coordinated and not
coordinated, joint projects with industry, contracted research etc.) Other outputs (e.g. research-education projects, PhD supervision,
quality of own scientific journal, events, post-graduate courses, spin-offs)
Negotiated Factors
Research mobility (e.g. in- and outgoing, affiliations abroad etc.)
Links with users (e.g. contracted research, other interactions with companies, market responsiveness)
Image building and ‚lobbying‘ (e.g. transfer of organisational standards, memberships, provison of expertise, PR, government committment towards the organisation)
Financial position (e.g. consistent funding or not, regular and returning clients, liquidity and cash flow management, financial management skills, assets for financial standing with banks, venture capital)
External Factors
Stage of transition of the S&T and economic system (e.g. RTDI policy, coherence among policies, attention paid to managerial aspects of RTDI, independence from political parties etc.)
Sectoral and national economy conditions (focus on food science) (e.g. capital market for financing of RTDI incl. FDI in RTDI, venture capital provision, promotion of long-term research programmes, demanding users and industrial conditions etc.)
Outputs
Evaluation methodology (incl. tailor-made qualitative and quantitative questionnaires)
SWOT based evaluation report (external evaluators) Action plan to improve the situation
(internal self-evaluation) (employee satisfaction evaluation) (auditing of the financial management) (assessment of technical infrastructure incl. IT)
SWOT results (page 1 of 3)
Strengths Weaknesses
Opportunities Threats
* No clear mission and strategy (yet)* Not enough critical mass in the current specialisations* Low number of students compared to the prevailing funding and promotion system* Relative lack of young staff* Low performing support staff* No accreditation of labs* Scarcely equipped and updated library
* Existing intrinsic value system towards scientific research* Increasing inter-disciplinary cooperation* A shared focus on food chemistry* Still sufficient number of researchers* Language skills and internationalisation of staff* Engaged and responsible self-governing management* Good team spirit
*Increasing future demand for food chemistry * UKIM’s intention to professionalise central management functions* More contracted research with industry through strategic partnership* More engagement in LLL* Development of joint study programmes with foreign partners
* Enlarged action space through European integration
* Danger of radically declining number of staff in near future* High insecurity about replacement of retiring personnel* Structurally caused inbreeding of personnel* Still limited management capacities at central levels* Low number of calls for research proposals at national and bilateral level* Low appreciation of RTD at UKIM and in Macedonia (ink. government)
Benchmarking
With whom to compare in order to assess if our data are good or not?
external benchmarking (best would be other IoCs) internal benchmarking (across time) functional benchmarking (analysis of functions and practices vis-a-vis
some accepted standards)
Due to the complexity, diversity and variability of RTO organisations and their environment, benchmarking produces rather signs of good practice than evidence of good performance!
Overall Assessment
1. Evaluation and benchmarking are powerful tools to generate evidence (versus myths), reflection (versus „doing business as usual), learning (what and how to improve), and an evidence-based starting point for interventions and change process
2. No size fits all, thus, many aspects have to be taken into account (see RECORD methodology). Different reference systems are needed for different levels.
3. It is seldomly used as policy tool, but rather bottom-up; shared responsibility might be powerful; independent external evaluators are recommended
4. It can be used for allocation of public money
5. If systematically applied, it will result in an improved system of innovation
6. A benchmark is not value-free, but requires „political“ legitimization.
Proposal for a joint Undertaking Establishment of a Regional RTDI Platform
Establishment as a (virtual) regional Centre of Competence „Owned“ jointly by ministries/agencies and specialised evaluation
institutes Supported by FP7 or SEE programm (higher start-up costs but low
running costs) Based on professional highest standards Provision of certified regional and international evaluators Provision of tools, templates and practices (e.g. ToR blueprints
etc.) Provision of tailor-made legal advice (e.g. national and European
public procurement laws) Provision of a necessary virtual infrastructure (e.g. access to ISI
Web of Knowledge, SCOPUS, Pascal; access to other information and data sources)
Functions of the Regional RTDI Platform (1)
1. Training of evaluators (setting of standards, certification) e.g. innovation surveys, input-output models, econometric models, cost
benefit analysis, control group approaches and quasi-experimentation, social network analysis, bibliometric analysis, patent analysis, foresight and technology assessment, qualitative methods like expert panels and peer reviews, focus groups, participatory evaluation, case study methodologies, logic chart approach etc.
project, programme, organisation and policy evaluations (ex-ante, interim, terminal and ex-post)
concept, design, process and impact evaluations
2. Training of policy-delivery systems (incl. agencies) training about best-suited evaluation designs for certain evaluation
questions information provision on the drafting of professional ToR for evaluation;
methods for technical and financial comparison and selection of offers seminars on national and European public procurement laws and their
implementation into practice
Functions of the Regional RTDI Platform (2)
3. Legal and Technical Advice inventory of good practices in the practical implementation of evaluation
procurement processes individual backstopping and advice for legal issues comparative study about public procurement rules and regulations in the
field of RTDI evaluation among member countries
4. Virtual Evaluation Infrastructure access to {expensive} data bases (such as ISI Web of Knowledge,
SCOPUS, patent statistics etc.) systematic provision of evaluation standards, evaluation reports,
handbooks and manuals, methodology briefs, templates and blueprints provision of an international evaluators database for programme,
organisation and policy evaluations continuous update of joint evaluators database for proposal evaluations
in certain fields of science
Functions of the Regional RTDI Platform (3)
5. Implementation of a series of programme evaluations (based on actual demand and variable geometry of the members of the RTDI Platform)
6. Benchmarking of RTD organisations (based on variable geometry) e.g. identification and assessment of potential Centres of Excellence e.g. benchmarking of RTD organisations in certain fields of science
(physics, ICT, agriculture, social science, humanities etc.) e.g. evaluation of agencies
7. Interface with other evaluation platforms (e.g. American Evaluation Association, DeGEval and others) e.g. through joint conferences e.g. through joint publications
Klaus Schuch
Zentrum for Soziale InnovationLinke Wienzeile 246
A - 1150 Vienna
Tel. ++43.1.4950442-32Fax. ++43.1.4950442-40
email: [email protected]://www.zsi.at
What do you think?