International League Tables:a new kind of game or more

of the same?

Dr Paul Greatrix, Registrar,The University of Nottingham

Dr Tony Rich, Registrar and Secretary,University of Essex

League Tables

• Background

• Who wants to know?• Who are the providers of this valuable information?

• Mad, bad and dangerous

• National and international dimensions

• Doing it yourself

• Measuring the unmeasurable?

Background

• The US experience

• Regulatory interest

• In the UK: The Times (1992)

• International dimensions

• They sell papers...• ...and, some would suggest, create perverse

incentives

Who wants to know?

• The Government

• The State

• The Funding Councils

• Potential Students

• Alumni

• Journalists

• Parents, teachers, advisors

• Employers - national and international

• Overseas sponsors

• Jo Public

• The Universities

• Dangerous obsessives

Who’s responsible for providing this valuable data?

• The Times

• Sunday Times

• The Guardian

• GUG

• Financial Times

• (Daily Telegraph)

• Shanghai Jiao Tong University

• Times Higher Ed (THE)

• Newsweek

• HEFCE

• others...

• and, of course, Government

Invaluable information

• For all stakeholders

• Intelligent decision-making

• Better than prejudice

• Reflecting the realities of the market place

• We have a right to know

• Employers have a right to know

• We aren’t stupid

League tables are a bad thing...

“The silly season that marks the publication of University league tables is nonsensical and illogical. As any New Scientist knows, letters into numbers; quality into quantity won’t go. League tables are simplistic, divisive and undermine the qualitative nature of a University’s work”

But we’ll use them anyway!

“Having said that, I’m not ashamed to report that we came a very creditable 79th overall, with my own department rating a particularly good score for research

- and as I remarked to the Dean, you can’t get much better than that.”

Dangerous...

• Criteria used do not reflect quality of education

• Historical data

• Variation over time

• Scores are institutional averages – mask strengths

• Distorting effect of weightings and scalings and data manipulation

• Many of criteria used are inter-related (ie not independent)

...extremely dangerous...

• Apples and elephants and paperclips

• Perverse incentives

• Hugely political – Government interest

• Open to manipulation

• Delivered by journalists

• Spurious precision – there’s no such thing as a good league table

• Serious consequences for universities, departments, staff and students

The Times 2007

1 Oxford2 Cambridge3 Imperial4 LSE5 St Andrews6 UCL7 Warwick8 Bristol9 Durham10 King’s

11 Bath12 Loughborough13 Edinburgh14 Southampton15 Aston16 York17 Exeter18 SOAS19 Nottingham20 UEA

Sunday Times 2007

1 Cambridge2 Oxford3 LSE4 Imperial5 UCL6 St Andrews7 Warwick8 York9 Bath10 Bristol

11 Durham12 King’s13 Loughborough14 Edinburgh15 Nottingham16 Southampton17 Exeter18 Lancaster19= Manchester19= Leicester

The Guardian 2007

1 Oxford2 Cambridge3 Imperial4 St Andrews5 UCL6 LSE7 Edinburgh8 Warwick9 Loughborough10 Bath

11 SOAS12 King’s13 Southampton14 Bristol15 York16 Manchester17 Durham18 Birmingham19 Nottingham20 Leeds

Good University Guide 2007

1 Cambridge2 Oxford3 Imperial4 LSE5 St Andrews6 UCL7 Bristol8 Warwick9 Bath10 Durham11 Loughborough

12 Aston13 Royal Holloway14= Nottingham14= York16 Edinburgh17= King’s17= Exeter19 Lancaster20= UEA20= Leicester20= Southampton

And a completely different approach:UEL: TQ in the South East 2005

Surplus of good practice over recommendations(in QAA audit reports)

1 East London 42 King's College London 23 Brunel 24 Queen Mary, London 05 Kent 06 Hertfordshire 07 Royal Holloway -18 London South Bank -39 Greenwich -610 Essex -611 Anglia Polytechnic -9

International value-added

• Higher Education is now a global business

• Global branding assisted by competitive ranking

• International benchmarking increasingly important, especially in research

• Student recruitment is increasingly international…

• …mobile students are increasingly choosy

• It’s all good healthy fun

Just as dangerous...

• Again, the criteria used do not reflect quality of education

• Archaic and irrelevant data

• Major biases towards large, English-speaking, research-intensive and science-focused universities

• Institutional scores are extraordinarily broad brush

• Distorting effect of weightings, scalings and data manipulation

• Far from comprehensive surveys of peers and employers

Times HigherWorld Rankings 2007

2= Cambridge (2)3= Oxford (3)5 Imperial College (13)9 University College London

(25)23 Edinburgh (33)24 King’s London (46)30 Manchester (40)37 Bristol (64)57 Warwick (73)59 London School of

Economics (17)

65 Birmingham (90)68 Sheffield (102)70 Nottingham (85)74 York (124)76 St Andrews (109)80= Leeds (121)80= Southampton (141)83 Glasgow (81)99 Cardiff (141)

Shanghai Jiao Tong 2007

4 Cambridge (2)10 Oxford (10)23 Imperial (23)25 UCL (26)48 Manchester (50)53 Edinburgh

(52)

62 Bristol (62)72 Sheffield (69)81 Nottingham

(79)83 King’s (83)92 Birmingham

(90)

Shanghai Jiao Tong 2007:Regional Analysis

Region Top 20

Top 100

Top 200

Top 300

Top 400

Top 500

North/ Latin America

17 58 98 138 164 197

Europe 2 34 80 123 172 208

Asia/ Pacific

1 9 24 42 64 100

Africas 1 2 5

Economist 2007Global MBA Rankings7 Cambridge: Judge10 Henley11 Cranfield15 London Business

School19 Ashridge27 Warwick31 Oxford: Said38 City: Cass50 Lancaster52 Leeds54 Aston

56 Edinburgh57 Manchester58 Imperial: Tanaka59 Durham62 Nottingham63 Birmingham64 Strathclyde65 Bath75 Glasgow88 Sheffield93 Bradford98 Newcastle

Financial Times 2008Global MBA Rankings

2 London Business School

10 Cambridge: Judge

19 Oxford: Said22= Lancaster

Manchester29 Warwick30= Cranfield

Strathclyde

35 Imperial: Tanaka41 City: Cass44 Edinburgh48= Leeds53 Bradford/

TiasNimbas69 Bath76 Nottingham

Another approach: Ecoledes Mines de Paris Rankings

1 Harvard10 Ecole des

Mines de Paris11 Oxford26 Manchester35= Queen’s35= Glasgow

60= Cambridge89= Heriot-Watt89= Bath89= Huddersfield89= Sheffield89= Uni of Wales:

Swansea(and over 100 others…)

Do it yourself

Your chance to show that you have a better understanding than the press of what really matters in HE

Do it yourself

All you have to do is use your skill and judgement to determine the most sensible weighting which should be applied to the criteria you select

UK approaches

• The Times– 8 indicators, z-scoring used– Teaching weighted @ 1.5 (now using only NSS)– Research weighted @ 1.5

• The Sunday Times– 9 indicators - student satisfaction weighted x1.5 (NSS);

teaching excellence x1.0– A level scores weighted x2.5 and research weighted x2– Also – Head Teachers’ assessment and drop-out rate

(variable)

UK approaches 2

• The Guardian– 7 indicators in subject tables (research ratings do not

feature)– Teaching quality (10%) and Student Feedback (5%) in

each subject table - both derived from NSS– Entry qualifications, SSR, Spend per student, job

prospects, Value-added @ 17%Overall table in 2007 is an average of subject tables

International approaches 1

Shanghai Jiao Tong: Ranking of World Universities

– 6 indicators covering: quality of education; quality of faculty; research output; performance relative to size

– Includes: Nobel and Fields winners among alumni and staff; highly cited researchers; articles in Nature and Science; articles in citation indices.

International approaches 2

Times Higher World University Rankings

– Peer review: 40%

– Citations per Faculty member and SSR: each 20%

– Recruiter review: 10%

– Proportions of international Faculty and students: each 5%

The critical factors(by frequency of appearance in the tables)

• Teaching evaluation: NSS

• A level scores

• Staff:Student Ratios

• Graduate employment rates

• RAE results

• Spending on library and computing

• Degree classifications

• Completion rates

An indicator too far(not covered in the UK tables - yet)

• Alumni giving

• Academic staff pay

• Research income

• Citations

• Brand impact

But international tables will increasingly influence methodologies of UK tables

What is to be done

• Publish prolifically and get top RAE grades• Be Highly Cited• Get articles in Nature and Science• Win a Nobel Prize or a Fields Medal• Make sure your students all get top jobs with big

multinationals (and then win Nobels)• Plug the institution relentlessly and cultivate peers,

Headmasters/mistresses and employers• Improve NSS results (and SSRs) - every little helps• Spend more on Library and IT and everything else to do with

teaching and learning• Recruit more international students and staff (all with Nobel

potential)

Conclusions

• They aren’t going to go away

• The international dimension will become increasingly significant

• Methodologies - for both national and international tables - are all dubious, at best

• They can and will be used by many different groups – but can be dangerous in the wrong hands

• Handle with great care!

Finally

For a copy of the presentation and further league table commentary and observations, visit:

http://registrarism.wordpress.com/