universities evaluations and rankings philippe vincke rector of the université libre de bruxelles
DESCRIPTION
Criticisms of the existing rankings (1) Competencies of the authors of the rankings Impossibility, for the reader, to reconstruct and verify the results (rankings are not« scientific ») No information about the goals, the intended uses, the aimed public Precise definition of « university » : are they all comparable ?TRANSCRIPT
UNIVERSITIES EVALUATIONS AND RANKINGS
Philippe VINCKERector of the Université Libre de Bruxelles
Evolution of higher Education
New Actors of higher education and research
Increasing mobility of students and researchers
Accountability of the universities, transparency
Evaluations, comparisons, rankings
Criticisms of the existing rankings (1)
Competencies of the authors of the rankings
Impossibility, for the reader, to reconstruct and verify the results (rankings are not« scientific »)
No information about the goals, the intended uses, the aimed public
Precise definition of « university » : are they all comparable ?
Choice of the criteria and of their relative importance
Research Education Costs Services Social aspects National context, legislation Financial ressources
Choice of the indicators Data validation
Criticisms of the existing rankings (2)
Criticisms of the existing rankings (3)
Bibliometry Quality of the data Discrimination among the scientific fields Different traditions (journals, books, proceedings,
number of authors, time span of valid research) Supremacy of the publications in English Which indicators ? (IF, citation index, h index, …)
Experts Do they exist ? How to choose them ? Which questions ? How to treat the answers ?
Numerical « manipulations » (1)
How is it possible to imagine that complex objects such as universities can be characterized by one number ?
The weighted mean can exclude good candidates
Example : A 41 97B 100 38C 68 68
Curious effects of normalization
(1) (2)A 2000 500B 1120 175C 400 370D 1600 45E 880 240F 160 435G 1360 110H 640 305
(1) (2) Global score
A 100 100 100B 56 35 45,5C 20 74 47D 80 9 44,5E 44 48 46F 8 87 47,5G 68 22 45H 32 61 46,5
RANKING : A, F, C, H, E,B,G,D
(1) (2)A 1600 500B 1120 175C 400 370D 1600 45E 880 240F 160 435G 1360 110H 640 305
(1) (2) Global score
A 100 100 100B 70 35 52,5C 25 74 49,5D 100 9 54,5E 55 48 51,5F 10 87 48,5G 85 22 53,5H 40 61 50,5
RANKING : A,D,G,B,E,H,C,F
Before : A, F, C, H, E, B, G, D
One modification of the score of A on one criterion.No change in the scores of the other universities
After : A, D, G, B, E, H, C, F
Inverse ranking !!
Other comments
Rankings are contested but used
Rankings have an influence on reality
Excesses are possible (financial bonus, or incitements,…)
Standardization effect
Conclusions
The rankings relayed by the media are not scientifically valid at this stage
Evaluation of research and higher education is a necessity
But it must be realized by competent people in the context of a clear policy and with explicit goals
There does not exist a unique method applicable in all institutions
Main questions (1)
Wich « objects » ?
Universities (definition?) Education programmes Diploma’s Research centers Research programmes …
Main questions (2)
What does one want to do ?
To compare To select the « best(s) » To rank To define « homogene » categories To detect strong and weak points To assign ressources …
Main questions (3)
For whom ? External autorities, government, … Potential partners (universities, research centers,
companies,…) External teachers or researchers Potential students Funding agencies Sponsors Public opinion, media Alumni Internal authorities Internal teachers or researchers Internal students …
Main questions (4)
For each « situation » (characterized by the answers to the 3 previous questions):
Which indicators ?
Quality of the data ?
Numerical treatment of the data !
Different approaches for different concrete questions
Choose the « best » education programme for this student ? (« best » for him)
Allocate financial resources to research centers Select the universities which could be « good »
partners for this company Identify the strong points of these universities for
students interested in studies in that field
Necessity of an interactive decision-aid toolbox for each possible user and question.