www.che.de
Multi-dimensional, field-based rankings
Gero Federkeil, CHE, Germany
Special Workshop: Introduction to Academic Rankings for the Rectors of Universities of the Republlic of Kazakhstan
Astana, 2009/06/13
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/132
Presentation
1.CHE – Centre for Higher Education Development
2.Rankings and information about higher education
3.The classical ranking-model
4.The CHE ranking approach
3DEAN Annual
Conference / Barcelona 16 -18 Nov. 2008
I. CHE – Center of Higher Education Development
private, not-profit organisation
founded in 1994 by Bertelsmann Foundation and German Rectors Conference
purpose: promotion of reforms in German higher education
Ranking of German universities among founding tasks of CHE; first ranking in 1998
activities:HE policy issues
consulting
ranking, since 1998
staff: ~ 30 people
more information: www.che.de
II. Users of rankings
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 4
(prospective) students: information about universities and programmes in the field the want to study
academics/researchers: comparison with colleagues in their field
rectors/university leaders : information about the position of their institution
policy makers: information about their national universities (international position, efficiency)
Diverse expectations / needs for information Rankings have to find a balance between those needs
incl. Giving information for users with different knowledge about higher education
5DEAN Annual
Conference / Barcelona 16 -18 Nov. 2008
III. The „classical“ model: ranking orthodoxy
There is a “classical” league table approach of rankings used by most rankings:
1. ranking of whole institutions
2. aggregation indicators into a single composite overall indicator by using fixed weights
3. league table with individual numerical positions (like soccer table)
6
Exampe: THES World Rankings
III. „Example: QS World Rankings
compositeoverall score
weights of indicators ?
But: is Johns Hopkins exactly 92,9 % as good
as Harvard?
league table with clear rank positions
ranking of whole universities
III .Critical remarks: ranking whole institutions
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/137
Example 1: Universities with identical score at a given indicator:
University A University BPsychology 37,8 30,0Sociology 15,5 27,0Economics 23,0 29,0Literature 17,6 25,0Mechanical Engineering 26,0 31,0Physics 25,5 28,4Chemistry 33,0 28,9Biology 37,0 33,0Medicine 45,3 Average 29,0 29,0
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/138
University A Field averagePsychology 37,8 32,0Sociology 15,5 16,0Economics 23,0 28,5Literature 17,6 15,0Mechanical Engineering 26,0 28,8Physics 25,5 32,1Chemistry 33,0 33,0Biology 37,0 41,0Medicine 45,3 50,5Average 29,0
III. Critical remarks: ranking whole institutions
Example 2: results in the context of the respective fields:
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 9
III. Critical remarks: composite indicators
U.S. News & World Report Ranking: Weights of indicators:
Indicator Weight
Reputation 25%
Retention Rate 20%
Faculty resources 20%
Student selectivity 15%
Financial resources 10%
Graduation rate 5%
Alumni giving 5%
Total 100 %
20%
30%
15%
5%
10%
15%
5%
100 %
But why not:
No individual ranks inleague tables
No overall score fromweighted indicators
No ranking of wholeuniversities
Multidimensionalranking
Ranking of single fields / programmes
Rank groups top intermediate bottom
IV. THE CHE approach – an alternative
labour market,employability
city, university
studentsstudy
outcome
teaching ressources
research
overall assessment(students,
professors)
internatio-nalisation
IV. The CHE-Ranking: Indicators
20 – 25 indicators ...
12ACA Policy Seminar, 4
April 2008
IV. The CHE-Ranking: Indicators
... from different data sources…
research
publications /citations (bibliometric analysis)
research grants (faculties/departments)
research reputation (professors survey)
13ACA Policy Seminar, 4
April 2008
IV. The CHE-Ranking:Indicators
... facts as well as judgements
teaching student-staff-ratio (fact)
student assessment of contact between students and professors
student assessment of course organisation
IV. CHE ranking: presentation of results
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 14
selecting a field ..
.... or a university
Looking at the results is possible either by ...
Step 1: Selecting a field
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 15
33 fields,
covering 80 % of German students
First overview:
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 16
5 selected indicators
Alphabetic list of universities
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 17
First overview:
Sort by indicator
Within groups: alphabetical order
- no league table!
Details for a single university : Humboldt Berlin
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 18
Facts as well as
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 19
subjective views by students
and professors(about reputation)
Details for a single university : Humboldt Berlin
Online: Interactive, personalised ranking
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 20
STEP 1:
Selection of (up to ) 5 indicators ...
... according to personal preferences
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 21
STEP 2: Decision about personal relevance of indicators
Interactive, personalised ranking
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 22
... and the result:
A personalised ranking
Interactive, personalised ranking
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 23
... which looks quite different if we select different indicators
Conclusions
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 24
Rankings should define their aims and target groups
... but they have different users anyway (students,
researchers etc.)
Rankings should adress the specific need for information of
different users
...which in most cases is about fields/subjects
... which differ with regard to the relevance of dimensions/
indicators (teaching, reasearch etc.)
Conclusions
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 25
So rankings should be
field-based in the first place
multi-dimensional,
showing the profile of institutions and
leaving the decision about the importance/
weight of indicators to users
And, last but not least, they should avoid giving
false impressions of exactness of league tables
Berlin Principles
Multi-dimensional rankings| Gero Federkeil | Astana 2009/06/13 26
15. Provide consumers with a clear understanding of all of the factors used to develop a ranking, and offer them a choice in how rankings are displayed.
Rankings should:
www.che.de
Thank you very much!
More information:
or
www.che.de/ranking
Or
www.
Special Workshop: Introduction to Academic Rankings for the Rectors of Universities of the Republlic of Kazakhstan
Astana, 2009/06/13