Research Assessment Exercise 2005(RAE2005)

University of Helsinki

Arto Mustajoki Based on the material of

Vice-Rector Marja Makarow

2

Assessment of university research

University of Helsinki (UH) University Senate’s decision RAE of UH every 6th year 1st time in 1999 (first in Finland) 2nd time in 2005

University of Tampere 2004 University of Jyväskylä 2005

Concept different from UK RAE Carried out by UK Council, every 8th year Peer review, no site visits Important financial consequences

3

Why a RAE at the UH?

Current challenges for universities in Finland Profiling Strategic spearheads of research International competitiveness Productivity progam Financial constraints

How to best meet the challenges Knowledge of strengths, weaknesses and potential External evaluation yields

solid objective data on quality of researchrecommendations for the future

4

What was evaluated?

Quality of research of departments Grade 1-7 (7 is best) Verbal arguments

Concepts of institutes, research networks and stations Only verbal arguments No grades

Interaction with society Only verbal arguments No grades

5

What was evaluated?1 Quality of research

Quality of research compared to that of SIMILAR

EUROPEAN INSTITUTIONS Expressed with grades from 1 to 7 Substantiated with verbal comments

NO comparison to results of UH REA 1999 NO comparison between UH departments

6

Target of assessment

All research performed in UH during 1999-2004

Units of assessment Faculty departments: 70 in 11 faculties Research institutes independent of faculties: 5

Individual researchers or research groups were NOT

evaluated

7

What was evaluated?2 Concepts

Verbal evaluation only of Research (field) stations: 5 Research networks: 5 Independent institutes: 5

Verbal evaluation of quality of research: Medical research programmes: 6

8

What was evaluated?3 The third task – interaction with society

Pilot project Legal obligation of universities since Aug 2005 Documentation of accomplishments not systematic

For example Expert tasks, popularization of science, text books,

clinical and commercial application of research data Units of assessment chose what to report Evaluators commented verbally The data serves to

Clarify how new knowledge has been tranferred to

be used in society Develop and document activities

9

How was the assessment carried out?

Peer review Documents on acitivities of 94 units of assessment

covering 1999-2004 via Evaluation Office to evaluators Desk work at home Preliminary drafting of Evaluation Report

Each panel one working week in Helsinki General info on Finnish science policy, university

system, UH Site visits to premises, infrastructure Interviews, researchers, PhD students, post-docs

Writing of Evaluation Report before leaving Finalizing of the Report by Panel Chair Editing the Report in Evaluation Office

10

Who were the evaluators?

148 mostly international scientific experts, in 21 panels From 21 countries 83% from Europe 9% from Finland 12% were there in 1999 18% from LERU universities 30% of panelists and chairs were women

Panelists were chosen from suggestions obtained from Scientific Council of UH Chairs of national Research Councils Rectors of LERU universities Intl top scientists

11

Research active staff (RAS) in 1999-2004

Researchers, post-docs and PhD students 4,000 annual work years Results of altogether 24,000 researchers’ work years was

evaluated

Number of RAS per unit of assessment: 3 - 640 Humanities and social sciences: 3-76 Natural science: 14-154 Medicine 24-640 Agriculture, Forestry, Bioscience, Pharmacy 7-340

12

Publications in 1999-2004

Only publications in the official data base of UH were

encluded in the assessment Altogether 60,000 publications

21,000 peer reviewed publications 22,000 other publications 2,400 monographies 2,000 PhD theses 10,000 popularized publications 600 text books

13

Competitive funding fetched by the researchers in 1999-2004 (kiloEUR)

Research Councils (Academy of Finland) 220.000 Ntl Technology Agency (Tekes) 61.000 Ministries 127.000 Ntl foundations 37.000 Intl foundations 27.000 EU FWPs 49.000

TOTAL 521 M€

14

Results1 Grades of the quality of research

Average 1999 4,66

Average 2005 5,8

15

Results3 Summary

From the 75 units 66 were evaluated also in 1999. From them

29 (44%) improved their grade 31 (47%) got the same grade 6 (9%) got a lower grade

20 units (27%) got the best grade 7 6 units improved from 4 to 6 1 unit improved from 3 to 6

16

Criteria of the grades

7: >50% of submitted works are at high intl level and all

others are at good intl level 6: >33% at high intl level and many others at good intl

level 5: >50% at least at good intl level and others at fair intl

level 4: >33% at good intl level and many others at fair intl level 3: >50% at least at fair intl level 2: >50% at fair intl level 1: none at fair intl level

17

Quality of research increased - why?

Performance of individual researchers and teams

Structural development at university level Concentration of activities to 4 campuses Fusion of departments (115>75)

Evaluation culture adopted

Recommendations of RAE1999 implemented

Background Sufficient national resources for research Intelligent national science policy

18

Principles of financial consequences of results

University Senate’s decision before publication of results

The best units and faculties are rewarded The resources will come from the university’s private

funds, not state budget No unit or faculty looses resources due to poor

performance

19

Rewards to the best units of assessment

Units of assessment 1.600 €/RAS/year will be awardes in 2007-2012

to units which obtained grade 7 1.600 €/RAS/year in 2007-2009

to units which improved to grade 6 from grade 3 or 4 27 units to be rewarded annually with

30.000 - 288.000 €

20

Rewards to the beast faculties

Faculties, rewarded during 2007-2009 6 faculties with the best average grade Amount of reward relative to number of RAS

after deduction of the RAS of units to be awarded

directly

6 faculties to be rewarded annually with 32.000 – 309.000 €

21

Total investment to quality research in 2007-2012

12 M€ to units of assessment 3 M€ to faculties

Total investment 15 M€

22

Evaluators’ comments to leadership

Structures which best support quality research in

universities Independent institutes Research programmes

Collaboration and strategic alliences Infrastructure and its sharing Proactive recruitment of researchers Funding of research Allocation of time for research Researchers’ careers Leadership

23

Governance of RAE2005Director of RAE2005 Vice-Rector for ResearchProf. Marja Makarow

Steering committee (Chair prof. M. Makarow) Prof. A. Mustajoki Director U. Mansikkamäki Mr. H. Kallasvaara

Panels and panelists: Steering committee >Reasearch Council of UH

ToR for evaluators & Guidance for units of assessment: Evaluation Office (K. Haila & R. Holm) > Steering

Committee > Reasearch Council of UH Principles of financial consequences

Research Council of UH Decision by University Senate

24

Publication of results

Duration of RAE procedure from May 2004 to March 2006

Site visits of panels in Helsinki May-June 2005: panels 1-4 September-November 2005: panels 5-21

Publication of results on the web on March 1, 2006 Summary Report (also available in printed fomat) Individual Evaluation Reports

www.helsinki.fi/research2005